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COMPUTER  NETWORK  RESEARCH 

Advanced  Research  Projects  Agency 
Semiannual  Technical  Report 

Decenber  31,  1971 


1.  INTRODUCTION 

Hns  semiannual  technical  report  covers  the  period  1  July  1971 
through  31  December  1971.  Our  activities  have  focused  on  nodeling,  analysis, 
measurements  and  systems  software.  In  Section  2  below  we  discuss  sore  of 
our  results  from  modeling  and  measurement  of  tine-shared  computer  systens; 
in  this  section  also  are  presented  results  from  the  network  modeling  and 
simulation  activities.  In  Section  3  we  describe  seme  of  our  network  measure¬ 
ment  activity,  including  proposed  specifications  for  additional  required 
tools.  In  Section  4  we  describe  some  of  our  network  and  systems  software 
development  progress. 

2.  ANALYTIC  STUDIES 

Modeling  and  Measurement  of  Time-Sharing  Sys  terra 

Our  study  of  resource  allocation  strategies  in  time-shared  computer 
systems  has  involved  complementary  analytic  and  measurement  efforts.  This 
section  sunmarizes  our  recent  work  in  these  areas. 

We  are  continuing  research  on  multiple  resource  queueing  models. 

A  Master's  thesis  entitled  "Carparison  of  Various  Queueing  Network  IfodeJs" 
ty  F.  Tabagi  was  completed  in  June  1971.  This  thesis  concerned  itself  with 
the  Gordon  and  Navel  1  model  of  closed  finite  queueing  systems  which  have  been 
found  appropriate  as  models  for  time-shared  and  multi -progranmed  oonputer 
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systems .  The  idea  was  to  take  the  recent  work  on  diffusion  approximations 
to  queueing  systans  and  to  attarpt  to  use  these  approximations  in  the  Gordon 
and  Newell  Model.  The  reason  for  this  attenpt  is  tiiat  the  Gordon  and  Newell 
solution  becomes  extremely  difficult  with  medium  size  and  large  systems. 

The  results  of  the  thesis  indicate  that  the  diffusion  approximation  is  indeed 
useful  if  it  is  applied  to  the  critical  (saturation)  node  in  the  network;  as 

a  by-pro  iuct  it  was  also  found  that  a  oanplete  deoanposition  using  methods 

* 

similar  to  Jackson's  model  is  also  an  excellent  candidate  for  approximating 
the  Gordon  and  Newell  systems. 

In  the  Gordon  and  Newell  Model  all  of  the  custoners  in  the  system 
are  required  to  be  statistically  identical.  For  example,  they  trust  all  have 
the  same  service  time  distribution  at  each  resource.  We  are  currently  investi¬ 
gating  networks  of  queues  models  in  which  each  customer  may  exhibit  distinct 
behavior. 

Our  work  on  time-sharing  scheduling  algorithms  is  continuing.  R.  Mintz 
will  present  a  paper  entitled  "Waiting  Time  Distribution  for  Round-Pobin  Queue¬ 
ing  Systems,"  which  solves  for  the  distribution  of  waiting  times  conditioned 
on  service  required  for  round-robin  scheduling  with  finite  quanta  at  the  XXII 
Polytechnic  Institute  of  Brooklyn  Symposium  on  Ccinpu ter -Communication  Networks 
and  Teletraffic.  Previous  results  gave  only  the  mean  value  of  waiting  time 
conditioned  on  service  required. 

Currently  implemented  and  presently  being  documented  is  a  time-shared 
scheduling  algorithm  simulator  prepared  by  Walter  Sheets  for  his  Master's 
thesis.  This  simulation  language  permits  one  to  synthesize  a  scheduling 
algorithm  and  then  simulate  the  behavi  jr  of  that  algorithm  with  artificially 

*"Networks  of  Waiting  Lines,"  Operations  Res.,  5:518-521  (1957)  . 
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generated  traffic.  The  program  is  extrronely  flexible  and  pemits  one  to 
easily  determine  the  gross  benavior  of  scteduling  algorithm  prior  to  any 


intense  analytical  effort. 

A  measurement  project  cn  the  SEX  timesharing  system  was  begun  by 
J.  Wong  as  his  Master's  thesis.  Part  of  the  thesis  was  devoted  to  the 
problem  Of  scheduling  I/O  requests  to  multiple  disks  cn  the  same  channel. 

An  optimal  algorithm  was  determined  for  such  a  configuration.  The  remainder 
of  the  thesis  concerned  memory  allocation.  The  SEX  system  attempts  to  preload 
the  working  set  of  a  process  when  it  is  to  run.  Measurements  showed  that 
interactive  processes  exhibited  poor  behavior  since  each  interacts  typically 
took  very  little  execution  time  and  required  a  change  in  the  working  set. 
System  calls  were  implemented  which  allcw  a  process  to  determine  its  own 
working  set.  The  executive  command  interpreter  was  irodifred  to  set  its  work¬ 
ing  set  as  soon  as  the  command  to  be  executed  is  decoded.  This  technique  sig¬ 
nificantly  reduced  page  faulting  for  this  process.  We  are  currently  investi¬ 
gate  the  general  notion  of  allying  a  process  to  advise  the  operating  system 


of  its  resource  requirements . 

During  this  reporting  period  an  extensive  study  of  program  behavior 
has  begun.  An  interpreter  has  been  constructed  fcr  the  Sigma-7  which  serves 
as  a  basic  tool  for  these  studies.  H.  Opderbeck  wrote  the  interpreter  and 
used  it  for  program  behavior  studies  for  his  Master's  thesis.  In  addition 
to  investigating  the  performances  of  knevn  paging  alcprithms,  he  also  defined 
a  new  class  of  paging  algorithms  and  studied  their  performance.  This  new 


class  of  algorithms  determines  the  number  of  pages  to  be  allocated  to  a  process 
cn  the  basis  of  its  current  page  fault  rate.  When  a  page  fault  occurs,  the 
new  page  may  replace  a  page  from  dm  process  or  be  allocated  as  an  additional 


page  for  the  process.  This  decision  is  made  on  the  basis  of  the  recent  page 
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fault  rate.  If  the  page  fault  rate  is  low,  the  memory  allocation  of  the 
process  is  periodically  reduced.  Measurements  have  shewn  that  the  alcpritlm 
to  cause  processes  with  very  different  addressing  behavior  to  run  with 
approximately  the  same  efficiency.  At  the  ACM/IEEE  Second  Symposium  on  Prob¬ 
lems  in  the  Optimization  of  Data  Ccrrtunications  Systems  (Palo  Alto,  Calif. , 

Oct.  20-22,  1971) ,  W.  W.  Chu  presented  a  paper  entitled  "Danultiplexing 
Considerations  for  Statistical  Multiplexors."  In  this  work  he  considered 
the  effect  of  statistical  multiplexors  on  buffer  occupancy  and  the  implica¬ 
tions  of  these  effects  cn  the  design  of  time-shared  scheduling  algorithms. 

This  paper  is  attached  as  Appendix  A. 

These  studies  sire  continuing,  particularly  in  investigating  the 
dynamic  performance  of  memory  allocation  and  paging  algorithms .  These  studies 
are  preliminary  to  the  development  of  models  of  program  behavior  and  studies 
of  mul ti-progranming  system  memory  allocation  strategies. 

2.2.  Model  ire;  and  Simulation  of  Computer  Networks 

A  paper  has  been  accepted  for  the  Spring  Joint  Computer  Conference 
1972  authored  by  H.  Frank,  R.  E.  Kahn  and  L.  Kleinrock  entitled  "Computer- 
Odtmunicaticn  Network  Design — Experience  with  Theory  and  Practice."  This 
paper  is  attached  as  Appendix  B  and  discusses  the  design  experience  for  the 
ARPA  Network.  It  is  a  rather  general  paper  and  contains  some  important  in¬ 
sights,  concepts  and  guidelines  for  design  of  networks. 

A  second  paper  authored  by  L.  Fratta,  M.  Gerla  and  L.  Kleinrock 
entitled  "The  Flow  Deviation  Method:  An  Approach  to  S tor e- and-Forwa rd  Communi¬ 
cation  Network  Design"  has  been  submitted  to  the  Networks  Journal  and  is 
attached  as  Appendix  C.  This  paper  discusses  a  heuristic  procedure  for  ob¬ 
taining  the  solution  of  the  routing  problem  and  of  the  capacity  assignment 
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problem  in  corputer  networks .  This  work  is  continuing  and  we  currently  have 
developed  an  exact  method  for  solving  these  problems  which  seens  to  be  rea¬ 
sonably  efficient.  It  has  been  implemented  in  FORTRAN  and  can  solve  a  21- 
node  net  in  about  30  seaonds  cpu  time  an  the  360/91  with  an  accuracy  of  10-^ 
in  the  minimum  delay  computation .  We  are  presently  attempting  to  reduce  this 
computation  time  by  improving  a  frequently  used  optimization  routine;  in  par¬ 
ticular,  we  are  acrrparing  the  performance  of  a  gradient  method,  a  parallel 
tangent  method,  and  a  Newton- Raphson  method.  Our  concentration  on  optimally 
selecting  capacities  from  a  discrete  set  is  continuing  and  we  are  currently 
working  on  a  subop timum  algorithm  in  this  regard. 

A  third  paper  by  G.  Cble  and  L.  Kleinrock  entitled  "An  Analysis  of 
the  Separation  Between  Packets  in  a  S tore-and-Porward  Network"  has  been 
accepted  for  the  MRI  International  Symposium  XXII  on  Cbnputer-Ccmmuni cations 
Networks  and  Teletraffic,  April  4-6,  1972.  In  this  paper  an  analysis  is  car¬ 
ried  out  for  the  inter-packet  spacing  which  may  be  expected  after  a  multi- 
packet  message  travels  through  a  nunber  of  nodes  cn  the  way  to  its  destination. 
This  computation  is  important  in  attempting  to  calculate  the  expected  tine 
that  a  reassanbly  buffer  will  be  occupied  while  awaiting  reception  of  a  multi¬ 
packet  message.  The  paper  is  attached  as  Appendix  D. 

In  conducting  research  for  his  Ph.D.,  Gary  Fultz  has  carried  out  the 
specification,  performance  verification  and  documentation  of  various  store- 
and-forward  computer  network  routing  algorithms  during  this  reporting  period. 
Three  major  classifications  of  routing  algorithms  have  been  investigated. 

a*  Deterministic  routing  strategies.  Deterministic  routing  algo¬ 
rithms  conpute  routes  based  upon  given  deterministic  decision  rules  and  pro¬ 
duce  loop-free  routes.  The  procedures  inresti gated  assumed  nultiple  priority 
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message  traffic,  a  given  network  traffic  matrix,  a  known  network  topology 
and  a  convex  network  cost  fmction  (generally  average  message  delay) .  Both 
optimal  and  sibcptimal  techniques  have  been  examined  which  attempt  to  find 
sets  of  routes  (and  henoe  the  network  traffic  flew)  which  minimizes  the  net¬ 
work  cost  function. 

A  nurber  of  subop timal  algorithms  were  formulated  which  gave  perform¬ 
ance  within  1%  of  that  foind  with  the  optimal  technique,  and  required  less 
progranming  effort  and  aonputation  time  than  the  optimal  technique.  The 
subop  timal  methods  are  useful  in  the  abstract  design  of  lew- cost  networks 
where  the  routing  algorithm  must  be  repeatedly  applied  during  the  optimization 
of  the  network  structure. 

The  results  of  the  optimal  technique  are  useful  as  both  a  bound  on 
the  performance  of  the  subop  timal  deterministic  routing  procedures  and  sto¬ 
chastic  routing  procedures  and  to  gain  understanding  about  the  deoorposition 
of  optimum  flows  in  the  network  in  terms  of  elanentary  flews  (a  special  form 
of  basis  for  the  description  of  any  arbitrary  network  traffic  flew) .  Experi- 
nental  results  have  shewn  that  it  is  relatively  easy  to  design  heuristic 
algorithm  structures  which  are  ocrputatiaial ly  simple  and  have  good  perform¬ 
ance. 

b.  strehastic  routing  strategies.  Stochastic  routing  algorithms 
operate  as  probabilistic  decision  rules  and  utilize  a  certain  amount  of  infor¬ 
mation  concemi ng  the  delays  within  the  network.  A  nutber  of  algorithms  have 
been  designed  and  their  performance  has  been  foind  using  simulation  techniques . 
This  class  of  algorithms  is  particularly  ^plicable  to  operational  networks. 

Of  the  algorithms  simulated,  most  of  than  gave  rise  to  reasonable 
values  of  ness  age  delay  as  the  network  traffic  load  was  varied;  hewever,  one 
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specific  algorithm  shewed  superior  performance  when  transmission  link  failures 
were  introduced  into  the  simulation. 

Further  study  is  required  to  assess  the  actual  conplexity  of  these 
algorithms  when  implemented  in  an  operational  network. 

c*  — P*  °°ntrol  strategies.  Flew  control  routing  strategies  are 

perhaps  a  ooirbi nation  of  both  deterministic  and  stochastic  techniques,  but  in 
addition  take  advanta^  of  the  operational  network  protocol  in  their  opera¬ 
tic...  The  major  objective  of  these  algorithms  is  to  provide  small  delay  times 
for  highly  interactive  messages,  and,  at  the  same  time,  provide  high  bandridth 
for  long  messages  or  for  file  transfer  between  renote  computers. 

Most  of  the  effort  to  date  in  this  area  has  been  concerned  with  a 
review  of  what  other  researchers  have  ordered,  and  special  attention  has 
been  plaoed  on  proposed  techniques  by  BHN.  The  major  effort  has  been  focused 
ipen  techniques  which  alleviate  high  congestion  points  within  tte  net  and 
efficient  storage  allocation  procedures  within  the  nodes  to  insure  the  accep¬ 
tance  of  arriving  messages. 

It  is  felt  that  flew  control  techniques  are  an  important  part  of  any 
operational  network  doctrine  and  must  be  included  in  the  design  of  the  basic 
network  routing  strategy  for  efficient  network  operation. 

3.  NETWOFK  tEflSUiEMENTS 

This  section  covers  our  activity  on  network  measurement  over  the  cur¬ 
rent  reporting  period.  Our  activity  in  this  field  has  been  increased  sig¬ 
nificantly  and  some  measurements  have  already  been  taken;  many  more  will  be 
taken  in  the  immediate  future. 
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3.1.  Experiments  Performed 


a.  A  set  of  measurements  of  the  subnet  was  performed  to  determine 
the  behavior  of  an  IMP  under  a  heavy  load.  The  measurements  were  performed 
by  using  the  message  generators  in  several  imp's  sending  similar  length  mes¬ 
sages  all  to  one  site's  discard  fake  Host.  One  of  the  sending  sites  was 
monitored  using  emulative  statistics  messages .  Three  quantities  were  calcu¬ 
lated:  avercige  rornd  trip  times  (message  -  KFNM  time) ,  average  throughput 
(including  retransmissions)  and  average  adjusted  throughput  (average  nunber 
of  round  trips  times  number  of  words/message) .  These  three  quantities  are 
plotted  as  a  function  of  the  nuiber  of  interfering  senders  in  Figures  1,  2 
and  3.  The  knee  of  each  curve  (where  all  senders  are  sending  multipacket 
messages)  indicates  that  there  is  roan  for  a  maximum  of  three  multipacket 

messages  in  the  reassembly  buffer  space.  This  is  cc  si  stent  with  what  we  know 
about  the  IMP  systems. 

b.  An  experiment  using  the  trace  package  in  the  IMP's  wa^  used 
successfully  in  helping  UCLA's  Campus  Oaiputing  Network  debug  the  network 
hard^are/software.  They  had  an  internal  trace  and  compared  it  to  the  trace 
we  get  from  the  IMP  and  in  doing  so  were  able  to  isolate  their  bug. 

3.2.  Tools  Built 

At  the  beginning  of  this  reporting  period  a  set  cl  assembly  language 
programs  existed  for  setting  IMP  parameters  and  collecting,  in  a  file,  mes¬ 
sages  from  the  IMP  statistics  package.  Also  two  FORTRAN  conpatible  subroutines 
existed  for  reading  these  files  so  that  reduction  of  the  data  could  be  per¬ 
formed  by  a  FORTRAN  program.  (Indeed,  sane  reduction  programs  were  written 
too.) 
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Number  of  Interfering  Senders 
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Figure  1 
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Figure  3 
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NOTES  ON  THE  "INTERFERING  TRAFFIC"  EXPERIMENT 


In  the  three  figures,  the  expression  "iWj"  is  defined  to  mean: 


The  measured  IMP  (UCLA)  is  sending  i-packet  messages 
and  the  interfering  senders  are  sending  j -packet  mes¬ 
sages.  All  messages  are  destined  for  RAND's  IMP  discard. 


Ttotal  throughput  is  measured  from  accumulated  statistics  and  represents  an 
average  of  2T siitples.  Accunulated  statistics  produce  a  count  of  the  total 
nvitber  of  words  sent  from  UCLA  to  RAND. 


Adjusted  throughput  is  oonputed  on  the  basis  of  total  data  words  sent,  and 
thus  excludes  leaclir  and  other  control  bits. 


itound  trip  times  represent  the  time  it  takes  for  a  message  to  be  delivered 
from  UCLA  to  RAND  and  for  the  associated  FENM  to  be  returned. 


As  the 
starts 
of  new 
ciated 


nutber  of  interfering  sites  increases,  a  new  IMP  traffic  generator 
to  send  nessages  to  RAND's  IMP  discard  program.  The  order  of  addition 
interfering  sites  is  fixed,  with  the  n^1  new  interfering  sender  asso- 
with  a  fixed  site  as  follcws: 


Ttotal  Nurber  of 
Interfering  Senders 

Next  Additional 
Site  Number 

0  — ► 

1  (UCLA) 

1  — ► 

5  (BBN) 

2  — * 

8  (SDC) 

3  — ► 

11  (Stanford) 

4  — ► 

9  (Harvard) 

5  —* 

4  (Utah) 

6  — ► 

16  (NASA-Ames) 

7  — ► 

6  (MIT) 

8  — ► 

12  (Illinois) 

12 
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SOE  OBSERVATIONS  CN  TOE  RESULTS 


1.  The  curve  labeled  "3"  in  adjusted  throughput  increases  at  4 
interfering  senders.  We  conjecture  that  since  UCLA  is  send¬ 
ing  1-packet  messages  and  the  interfering  senders  are  sending 
8-packet  messages ,  the  multipacket  packet  senders  interfere 
so  severely  with  cne  another  while  contending  for  reassembly 
buffer  space,  that  more  single-packet  messages  can  be  sent  to 
discard.  This  may  be  a  false  conjecture,  since  the  same  effect 
is  not  seen  when  the  multipackets  are  of  length  4. 

2.  Comparing  curves  6  and  9  in  adjusted  throughput,  it  appears 
that  against  the  same  interference  (i.e.  8-packet  messages) , 
it  is  better  to  send  longer  messages  (i.e.  8  instead  of  4) 
unless  you  can  send  single-packet  messages  (i.e.  curve  3)  . 

3.  When  UCLA  is  sending  multipacket  messages  and  interfering 
senders  are  sending  multipacket  messages,  delays  jimp  at  3 
interfering  senders  because  there  is  buffer  space  (24  packets) 
for  only  3  messages. 

4.  When  interfering  sites  are  sending  single  packets  while  UCLA 
sencfe multipackets,  there  isn't  much  throughput  loss  due  to 
interference. 

5.  Similarly,  if  UCLA  sends  single-packet  messages,  multipacket 
messages  do  not  interfere  very  much  (curves  1,  2,  3) . 
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The  above  set  of  programs  were  augmented  by  several  FORTRAN  compat¬ 
ible  subroutines  for  setting  the  state  of  IMP  parameters,  so  that  no?  an 
experiment  may  be  run  under  the  control  of  a  FORTRAN  program  running  under 
the  time  sharing  system.  (In  fact,  the  first  experiment  discussed  in  3.1 
was  performed  in  this  fashion.) 


3.3.  Tools  Specified 

A  large  part  of  our  effort  during  this  reporting  period  was  to 
identify  needed  tools  and  to  begin  careful  specifications  of  each.  In  so 
doing,  we  have  categorized  our  existing  tools  as  well  as  the  nw  required 


ones. 


Among  the  existing  tools,  we  have  the  following: 

1.  Net  statistics  gatherer 

a.  enables/disables  IMP  background  programs  via  Host 
originated  messages  to  parameter  change 

b.  collects  resulting  statistics  messages  sent  to  Host 
by  background  programs  (cumulative  statistics,  traces, 
snapshots) 

c.  runs  under  SEX  (a  less  powerful  version,  one  requiring 
IMP  tty  interaction  for  parameter  changes,  runs  under 
RAD  75) 

2.  Net  statistics  formatter 

a.  processes  statistics  message  file(s)  collected  by  (1) 

b.  generates  tabular  reports 

c.  runs  under  SEX  (also  RAD  75  as  a  part  of  the  Standalone 
measurement  program) 

3.  Host  traffic  generator 

a.  generates  HWM  driven,  fixed  or  randan  length  messages 
over  one  or  more  links 

b.  generates  randan  interval,  fixed  or  random  length  messages 
over  one  or  more  links 

c.  runs  under  RAD  75  as  part  of  the  Standalone  Network 
measurement  program 

4.  INF  traffic  generator 

a.  generates  RENM  driven,  fixed  length  messages  on  one  link 

b.  runs  as  background  program  in  IMP 
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5.  Network  status 

a.  shews  local  connection  status  (open,  closed,  wait,  etc.) 

b.  shews  what  Hosts  ore  up  (Hosts  fran  which  the  NCI? 
received  a  reset  or  a  reset  reply) 

c.  runs  under  SEX 

6.  IMP  facilities 

a.  DEBUG  -  allows  examination  of  core  locations 

b.  PARAMETER-CHANCE  -  controls  background  programs 

c.  DISCARD  -  returns  RFNM  and  throws  message  away 

d.  IMP  tty  -  usable  for  interaction  with  EEBUG  &  PARAMETER 
CHANGE  if  HOST  is  dead 

Among  the  projected  tools  are: 

1.  Network  survey  (almost  operational  now) 

a.  reports  Host  status  -  (NCP  tp/down) 

b.  reports  connection  status  -  what  other  Host  communicating 
with  Host 

c.  reports  Host  resource  status  -  what  software  is  available 

2.  Remote  Host  traffic  generator 

a.  generates  KFNM  driven  or  randan  interval,  fixed  or  random 
length  traffic  an  one  or  more  links 

b.  runs  at  remote  Host  under  time-sharing,  as  a  part  of  a 
Network  handler,  or  standalone.  Controllable  from  UCLA. 

3.  Remote  Host  traffic  reflector 

a .  receives  message 

b.  time  tags  it 

c.  sends  it  back  or  cycles  address  sequence 

4.  Remote  Host  discard 

a.  receives  message 

b.  time  tags  it 

c.  saves  times  for  later  processing  at  remote  Host  or  trans¬ 
mission  back  to  us 

5.  Remote  Host  statistics  gatherer 

a.  performs  functions  of  our  Net  statistics  gatherer 

6.  NCP  statistics  hooks 

a.  reports  amount  of  traffic  over  each  connection 

7.  Ability  of  IMP  to  change  message  lengths  to  s inula te  higher  and 
lower  speed  lines. 
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So  far  we  have  given  in  SPATE  Design  Note  #94  the  careful  specifica¬ 
tion  for  Host  artificial  traffic  generator.  It  is  as  follows: 


SPADE  Design  Note  #94 


V.  Cerf 
L.  Nelson 
A.  Ollikainen 
20  December  71 


Specifications  for  a  Host  artificial  traffic  generator  essential 
features : 

1.  Transmit/receive  Network  messages  utilizing  level  one  protocol 
(Host-IMP) 

2.  Dynamic  modification  of  any  or  all  of  the  following  parameters 
according  to  some  distribution  or  density  function: 

a.  period  of  message  (inter-arrival  time) 

b.  Length  of  message 

c.  destination  of  message 

d.  link  #  of  message 

e .  message  type 

Functions  necessary  for  implementation  of  foregoing  features  (a  description 
of  how  UCLA's  traffic  generator  works)  : 

1.  Programmable  clock  routine,  driven  by  an  event  list,  schedules 
next  message  to  go  on  each  active  link.  Minimum  required  reso¬ 
lution  of  the  clock  routine  is  the  minimum  I3NM  return  time  on 
a  given  link. 

2.  Interrupt  routine  observes  the  state  of  each  link  and  decides 
whether  to  schedule  a  new  message  on  the  future  event  list 
depending  on: 

a.  when  IWM's  are  received 

b.  when  clock  interrupt  occurs 

3.  Lists  of  destinations,  periods,  lengths  and  message  types. 

4.  Accumulation  of  actual  traffic  to  each  destination. 


Rxneriment  traffic  parameter  specification  (an  artificial  traffic  generator 
SSdS  general  eSujh  to  allow  the  following  tyfes  of  traffic  specifications) 


1.  Destination  specific 

a.  for  each  destination — specify  inter-arrival  density  function 
b*.  for  each  destination— specify  length  density  function 
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(Requires  2N  density  functions  for  N  destinations  and  does  not  specify 
message  types ,  contents ,  etc. ) 

2.  Probabilistic  destination 

a.  specify  density  function  of  traffic  fraction  destined 
for  each  Host 

b.  same  as  in  (1)  (2  density  functions:  inter-arrival  tune 
and  length) 

(This  requires  only  3  density  functions.  Again  message  type  is  left  out.) 

3 .  Message  types 

a .  all  same 

b.  message  type  for  each  destination 

c.  message  type  density  function 

d.  message  type  density  function  for  each  destination 
(We  guess  that  (a)  and  (b)  are  sufficient  for  most  experiments.) 

4 .  Experiment  duration 

a.  specify  duration  initially 

b.  send  s tart/stop  messages  to  generator(s) 

Experimental  feedback: 

1.  Fbr  each  destination,  accumulate 

a.  total  bits  sent 

b.  total  messages 

c.  experiment  time 

2.  Record  time  waiting  after  IMP  says  "not  yet" 

3.  Record  nutber  of  "link  table  full"  messages  received 
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A  Ph.D.  thesis  by  Gerald  Cole  entitled  "Conputer  Network  Measure¬ 
ments:  Techniques  and  Experiments"  was  oonpleted  in  June  1971.  Hie  results 
of  that  study  have  be  in  used  as  points  of  departure  for  our  recent  measure¬ 
ment  activities  and  has  formed  the  foundation  for  this  work.  He  presented 
the  paper,  "Performance  Measurements  on  the  ARPA  Computer  Network,"  at  the 
ACh'/IEEE  Second  Symposiim  cn  Problems  in  the  Optimization  of  Data  Communica¬ 
tions  Systems,  Palo  Alto,  Calif.,  Oct.  20-22,  1971,  and  is  included  here  as 
Appendix  E. 

As  we  have  mentioned  above,  measurements  have  begun,  new  tools  have 
been  partly  specified,  and  many  new  ones  have  been  identified.  We  are  in 
the  process  of  oonpleting  the  specifications  cn  this  set  of  tools,  and  we  are 
finding  cooperative  Hosts  who  will  permit  various  of  these  tools  to  be  imple¬ 
mented  at  their  sites  cor  the  purpose  of  our  extensive  measurement  program 
which  is  now  well  under  way. 

4.  NETWOFK  AND  SYSTEMS  SOFTWARE 

This  section  covers  work  done  by  the  SPACE  Group  which  is  under  the 
leadership  of  Jon  Postel.  This  group  is  responsible  for  maintaining  and 
extending  the  SEX  time-sharing  system  and  the  development  of  network  software. 

4.1.  Network  Progress 

Jen  Postel  has  been  appointed  a  Network  Facilitator  by  ARPA.  In 
this  capacity  he  is  responsible  for  aiding  in  the  development  of  new  applica¬ 
tions  for  the  network,  new  protocols,  and  generally  being  of  assistance  to 
members  of  the  ARPA  Network  community.  He  has,  for  example,  consulted  with 
the  programmers  at  Tinker  APB  cn  protocol  and  program  construction  and  the 
Center  for  Computer-Based  Behavioral  Studies  at  UCLA  on  the  design  considera- 
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tions  and  cost  of  interfacing  with  the  ARPANET.  He  regularly  attends  Net¬ 
work  Working  Grra*  meetings,  has  led  discussion  gicp*  ate  has  been  very 
active,  particularly  in  the  defining  of  network  protocol. 

Existing  network  software  (NCP,  telnet,  LOGGER)  have  been  kept  up  to 
date  with  current  network  specifications.  Sene  ateitional  user  features 
have  been  added  to  these  processes  which  are  not  network  standard,  for 
example,  the  NCP  will  automatically  time-ait  a  connection.  We  have  also  been 
active  in  the  develc*ment  of  software  for  use  of  network  facilities.  Primary 
is  RJSNET,  a  program  white  provides  ^  ^  side  Qf  ^  m/n 

terete  Job  Entry  Service.  Another  exanple  is  fXFER,  a  program  which  provides 
access  to  the  USSR  360/75  Siaple  Minded  Pile  Service.  This  service  has 
proved  quite  useful  in  providing  quickly  ateessible  file  backup.  We  are  cur- 

"nUV  °°0Peratln9  With  ®*D  «»  «■  in  ^  development  of  a  Data  feconfigur- 
ation  Service  Qnpiler  which  will  provide  convenient  mans  for  reformatting 
files. 


4 -2.  System  Envelopment 

The  SEX  time-sharing  system  is  new  available  for  standard  user  serv¬ 
ice  70  hours  per  week.  Improvements  continue  to  be  made  to  the  systen,  such 
as  an  improved  text  file  manipulation  service  and  extending  of  graphics 
facilities.  Much  effort  has  gone  into  providing  measurement  routines.  We 
are  currently  implementing  a  new  disk  scheduler  to  cope  with  the  problem  of 
multiple  disks  on  one  channel.  We  are  also  experimenting  with  new  resource 
allocation  algorithm.  This  work  is  closely  tied  with  our  research  efforts 
in  computer  systan  modeling. 
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5. 


CONCLUSIONS 


Our  activities  then  have  concentrated  on  modeling,  amlysis  arx3 
measurement  both  of  time-shared  computer  systems  and  of  conputer  networks. 

The  AFPA  Network  of  course  forms  the  basic  model  in  our  network  studies.  Our 


network  and  systems  software  has  progressed  very  nicely.  During  the  next 
semiannual  period  we  expect  to  have  specified  additional  measurement  tools 
and  to  have  exercised  them  in  measuring  and  ionitoring  network  activity  and 
performance. 


5.1*  Publications  and  Presentations 

During  this  period  the  following  papers  were  published  or  submitted 
for  publication: 

Cantor,  D.,  "On  Non-Blocking  Sv itching  Networks,"  to  be  published  in  Networks. 

Chu,  W.  W.,  "Demultiplexing  Considerations  for  Statistical  Multiplexors," 

Proc.  of  the  ACM/IEEE  Seoond  Symposium  on  Problems  in  the  Optimiza¬ 
tion  of  Data  Ccmnunications  Systems,"  Palo  Alto,  calif.  Oct. 


Cole,  G.  D. ,  "Performance  Measurements  on  the  ARPA  Oomputer  Network,"  Proc. 
of  the  AOVIEEE  Second  Synposiun  on  Problems  in  the  OptimizatiorTo? 
Data  Caimunicaticns  Systems,"  Palo  Alto.  Calif.,  net.  IqT 

Fratta,  L.,  Gerla,  M. ,  and  Kleinrock,  L.,  "The  Flow  Deviation  ftethod:  An 
Approach  to  S tore-and-Forward  Oormunication  Network  Design,"  to  be 
published  in  Networks . 

Fbltz,  G. ,  and  Kleinrock,  L. ,  "Adaptive  Routing  Techniques  for  Store-and- 

Forward  Cbnputer-Ccrmuni cation  Networks,"  Proc.  of  the  tfrf  convention 
on  Oomnunications ,  Montreal,  Canada,  June  14-16,  1971,  pp.  39-1  to 
39-8. 

Kleinrock,  L.,  Muntz,  and  Hsu,  J. ,  "Tight  Bounds  on  the  Average  Response 

Time  for  Time-Shared  Oonputer  Systems,"  Proc.  of  the  1971  International 
Federation  for  Information  Processing,  Ljubljana,  Yugoslavia,  Aua.  23- 
28,  1971,  TA2,  pp.  50-58.  ' 

Kleinrock,  L.,  "A  Selected  Menu  of  Analytic  Results  for  Time-Shared  Oonputer 
Systems,"  to  be  published  in  Elektronische  Rechenanlagen. 

Kleinrock,  L. ,  and  Muntz,  R.  R. ,  "Processor-Sharing  Models  of  Mixed  Schedul¬ 
ing  Disciplines  for  Time-Shared  Systems,"  to  be  published  in  the  JACM. 
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The  following  presentations  were  given: 

Kleinrock,  L. ,  "Sone  ODrputer  Network  Models,"  Gxputer  Science  Department, 
University  of  London,  June  30,  1971. 

Kleinrock,  L. ,  "The  AFPA  Computer  Network,"  IBM  Research  Laboratories, 

Zurich,  Switzerland,  July  12,  1971. 

Kleinrock,  L. ,  Chairman  of  the  session  "System  Modeling,  Evaluation,  Cptimi- 
zation,"  and  banquet  speaker  at  the  Second  Symposium  on  Problems  in 
the  Optimization  of  Data  Ccrtnuxiications  Systems,  Palo  Alto,  Calif., 
Oct.  20-22,  1971. 

Muntz,  R.  R. ,  "Operating  Sy stars — A  User's  View,"  presented  at  the  Associa¬ 
tion  for  Computing  Machinery  Professional  Development  Seminar, 

August  1971. 

Muntz,  R.  R. ,  and  Wong,  J. ,  "Prooess-to-Scheduler  Comnuiication  to  Aid  Memory 
Management,"  presented  at  the  Workshop  on  System  Performance  Evalua¬ 
tion,  Argonne  National  Laboratories,  Oct.  6-7,  1971. 

Muntz,  R.  R. ,  and  Wong  J. ,  "A  Solution  to  the  Mil ti -shaft  Problem,"  XE6 
Users'  Group  Meeting,  Las  Vegas,  Nev. ,  Nov.  18-20,  1971. 
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DEMULTIPLEXING  CONSIDERATIONS  FOR  STATISTICAL  MULTIPLEXORS* 


Wesley  W.  Cliu 
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Abstract 


Demultiplexing  serves  as  an  Important 
function  for  statistical  multiplexors.  Its  pur¬ 
pose  is  to  reassemble  the  received  message  and 
distribute  it  tc  the  appropriate  destination.  The 
demultiplexing  buffer  can  be  modeled  as  a  finite 
waiting  room  queuing  model  with  batch  Poisson 
arrivals  and  multiple  distinct  constant  servers. 
Simulation  method  is  used  to  study  the  buffer 
behavior  for  the  uniform,  linear,  step,  and 
geometric  traffic  destination  functions.  The 
relationships  among  buffer  overflow  proba¬ 
bility,  buffer  size,  traffic  intensity,  average 
message  length,  and  message  destination  are 
presented  in  graphs  to  provide  a  guide  in  the 
design  of  demultiplexing  buffer.  Simulation 
results  reveal  that  buffer  input  message  which 
have  short  average  message  length  and  uniform 
traffic  destination  yield  best  buffer  behavior. 
Thus,  in  planning  the  CPU  scheduling  algorithm 
and  in  selecting  the  demultiplexing  output  rates, 
the  computer  communications  system  that  uses 
the  statistical  multiplexing  technique  should  also 
consider  the  desired  output  statistics  to  achieve 
optimal  demultiplexing  performance. 

I.  Introduction 

To  increase  information  processing  capa¬ 
bility  and  to  share  computer  resources, 
remotely  located  computers  and/or  terminals 
are  connected  together  by  communication  links. 
Many  such  computer  communication  systems  as 
time  sharing  systems  and  distributed  computer 
systems  are  already  in  existence  and  in  opera¬ 
tion.  The  additional  cost  to  such  systems  is 
the  communications  cost.  In  many  cases,  the 
communication  cost  is  a  significant  portion  of 
the  total  operating  cost.  To  increase  channel 
utilization  and  reduce  communication  cost, 
asynchronous  time  division  multiplexing  has 
been  proposed  for  data  communications.  1  The 
design  considerations  of  such  a  multiplexing 
(statistical  multiplexing)  system  has  been  ' 
reported  in  recent  literature. 2-6  Hore,  we 
study  tho  design  considerations  of  the 

*Thla  research  was  supported  by  the  U.S.  Office 
of  Naval  Research,  Research  Program  Office, 
Contract  No.  N00014-69-A -0200-4027, 

NR  048-129  and  the  Advanced  Research 
Projects  Agency  of  the  Department  of 
Defense  Contract  No.  DAHC  15-60-C-02B5. 


asynchronous  time  division  do  uultipl.xnig 
system  which  Is  an  integral  part  of  the  statistical 
multiplexor  design.  In  tin:  multiplexing  ease, 
the  outputs  from  tin;  system  going  to  a 
Single  destination,  for  i  xample,  a  computer, 
while  the  outputs  from  the  demultiplexing  system 
are  going  to  many  dr  .i...  .lions,  such  us  different 
users  mul/or  compute!  i  as  shown  in  Fig.  1. 


(a)  for  o  Imi.c  .l.r.nng  wy.lem 


(b)  for  0  diftlr.li.ilcU  Corr.pultr  lyilem 


0.  u*«r  teini.nul 
0  i  tlolitlicol  dtmuil.|.ieiur 
50  ■  llolitllcol  mullipitftor 
0  ■  compultr 
ED ,  l"  computer 

Fig.  1  Statistical  Multiplexing  Systems 

The  demultiplexor  may  consist  of  u  buffer 
and  a  switching  circuit.  The  input  truffle  to  the 
buffer  consists  of  strings  of  characters  (bursts). 
Tho  switcliing  circuit  distributes  tho  output  from 
the  buffer  to  tho  appropriate  destinations  aero  ril¬ 
ing  to  the  designation  in  tho  message  address, 
Tlius,  tho  demulliploxor  performance  is  strongly 
influenced  by  the  buffer  behavior  wldch  depends 
on  the  buffer  input  traffic  and  its  destination 
characteristics.  A  queuing  model  with  a  llnlto 
waiting  room,  batch  Poisson  arrivals,  and 
multiple  diitinot  constant  outputs  (Fig.  2)  is 
used  to  study  the  buffer  behavior. 
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KiC.  2  A  Model  for  Demultiplexing  Buffer 


The  complexity  "f  tim  il»multlplexlnf>  buffer 
makes  exact  an  .Iv  .la  of  sueh  ft  model  mc.thcmat- 
Ictilly  Inlr.ictnbl' .  Tlirrrfore,  computer  stmu- 
lfttion  lias  I  icon  1 1  •  ■  •  1  lo  study  the  rolntionshipft 
among  mi's  uijji'  ill  s' in  il  Ion  function,  nveinge 
traffic  level,  nr Imrnt  length,  overflow 
probability  (the  M  i.i'lfou  oi  i lie  total  number  of 
messages  r*'jei'*'’'l  I*'/  tti*’  lull  lor),  ami  buffer 
Hi7.c.  There  I'd  'I1""  '  ! i • 1  nr**  portrnycd  in 
graphs  anil  are  liopocl.o't  in  designing  dciniilti- 
plexiug  sy ut ruin  md  .••tore "iunl -forward  computer 
networks. 


II.  Analyst  :  of  |ii  ■iiuhipli'xlng  lliiBcr 
The  input  mess,*  i  to  the  buffer  can  be 
represented  by  Hu-.*  messnge 

arrivals,  messar''  1  •  1 1 1 ; t •  • ' .  and  . i,i.<ge  dost!” 

nations.  Those  . . i.  i  ..  ,.|  .  Intimately 

relatml  to  Hie  inifb  r  I ••  •  In v in r .  Ill  older  to 
describe  til.  input  mm  s  e'en  eel  vine,  at  tl.c  buf¬ 
fer,  three  r.in<l"in  n"'ebei  geiiecatm  s  are  iimuI: 

a  which  correspond  .  no  a  *:ir**  Intel" -arrival 

times,  /!  which . .  to  the  nunihcr  of 

characters  In  tin*  no  •• '  ,  •'"'l  T  which  corre¬ 
sponds  to  the  dest nod io,i  ..r  Hie  message.  In 
the  simulation  inodei,  the  nu  .e.ugo  lutor-ari  lval 
times  o  arc  assnnesl  to  he  exponentially  dis¬ 
tributed,*  and  the  rimming.-  lengths  f I  arc 

*In  the  distributed  i  onipnler  network  as  shown  in 
Fig.  ib,  the  traffic  input  to  the  demultiplexor 
is  the  output  from  the  multiplexor.  In  some 
cases,  the  traffic  output  from  the  multiplexor 
eun  be  approximated  an  Poisson  distributed. 
Should  the  multiplexor  output  become  very  dif¬ 
ferent  from  Poisson,7  then  the  actual  message 
inter-arrlval-tlme  statistics  should  be  used  in 
the  simulation. 


assumed  to  be  geometrically  distributed.  The 
destination  distribution  Is  transformed  from  the 
destination  function  as  discussed  in  the  follow¬ 
ing: 

The  destination  function  describes  trnffie 
Intensities  for  the  set  of  m  destinations;  that  is, 

f, (1)  ■  P.  1-1,2,  ...,m  (U 

where  Is  the  traffic  intensity  for  the  1th 
destination. 

The  message  destination  function  for  a 
given  set  of  users  depends  on  their  applications 
and  should  be  derived  from  measured  statistics. 

In  order  to  perform  random  sampling  on 
traffic  destination,  we  need  to  transform  fd(i) 
into  a  message  destination  distribution,  f^O). 
This  transformation  ean  be  performed  by  nor- 
malltlng  Eq.  (1)  to  a  probability  destination 
function;  that  is, 

/m 

Z  p,  1“1*2 . m  (z) 

j-1  1 

Thua, 


m 

Z  - 1 

i=i  7 

where  m  is  the  total  number  of  message 
destinations. 

A  set  of  random  numbers,  {C^,  5^}, 

corresponding  to  random  variables  a,  fi ,  and  y 
arc  generated  to  represent  a  message  arriving 
at  the  demultiplexing  system.  When  u  message 
arrives  at  the  buffer,  two  operations  take  place: 
First,  the  status  of  the  designated  facility  is 
interrogated.  If  the  facility  is  busy  and  the 
buffer  is  not  full,  the  burst  enters  the  buffer 
and  Is  concatenated  with  the  queue  of  characters 
walling  for  that  facility.  If  the  facility  is  idle, 
and  If  the  buffer  is  not  full,  the  first  character 
of  the  burst  is  sent  to  the  facility  while  the 
remaining  characters  enter  into  the  buffer  and 
output  at  euch  subsequent  clock  time.  Second, 
the  contents  of  the  register  that  keep  track  of 
the  total  length  of  the  buffer  Is  updated.  Because 
of  distinct  destinations,  the  total  volume  of 
output  from  the  buffer  varies  with  a,  ft,  and  y. 
The  output  from  the  demultiplexing  buffer 
depends  on  both  the  number  of  characters  in 
the  buffer  and  their  destinations.  The  simula¬ 
tion  prngrum  keeps  a  reeord  of  the  number  of 
cJnirwlers  in  the  buffer  at  the  beginning  of  each 
message  service  intorval.  When  the  length  of  nn 
arriving  message  exceeds  the  unoceuplcd  storage 
space  of  the  buffer,  then  a  buffer  overflow  event 
has  occurred.  Tho  frequency  of  occurrence  of 
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such  on  overflow  event  gives  the  estimate  of 
buffer  overflow  probability,  PQ/.  A  flow  chart 
of  the  simulation  program  Is  shown  in  Fig.  3. 


'  Tm  Ml*  tnMllM  mi  Ina  Me 

Ml v  km  a*«'M  !<•*  tm  CM  nmMirnt. 


Fig.  3  A  Logic  Flow  Chart  for  the 
Buffer  Simulation  Model 


One  of  the  moat  important  parameters 
that  deacrlbea  the  traffic  congestion  of  the 
demultiplexing  ayatem  la  traffic  Intensity.  Since 
the  output  meaaagea  from  the  buffer  are  sent  to 
various  destinations,  the  traffic  intensity  of 
each  destination  would  be  different  and  would 
equal 

M. 


"1 


1  1 
**1 


(3) 


where 

A  *  message  arrival  rate  for  the  1^ 
destinations 

■  average  message  length  for  the  Ith 
destination 

u  ■  transmission  rate  for  the  l*h 
destination. 


The  average  traffic  level,  p",  for  the 
demultiplexing  system  is  the  average  of  the  traf¬ 
fic  Intensities  of  the  m  destinations:  that  Is, 


To  study  the  effect  of  destination  distributions 
on  buffer  behavior,  five  types  of  destination 
functions  ure  used  In  tiie  simulation  model  us 
shown  In  Fig.  4.  The  relationships  among  P0f, 
buffer  sizes,  und  destination  distributions  for 
selected  £'s  und  p's  ure  obtained  from  simu¬ 
lation  and  portrayed  In  Figs.  !i  und  li. 


«u  i 


Tins  'l>  tii  utlon  functions  for 
ji  ■  O.i,  1.  r  tin;  same  forms 
a.;  l l..it  , a  0,  It  except  the 
Pl'  i  ui  >  r.  die  ed  by  u  tuctor 
of  1! u 


Fig.  4  Various  l^pi-s  nt  Di'uiiii.itlon  Functions 


(4) 


Fig.  6  Buffer  Overflow  Probability  va  Buffer 
Size  for  Average  Traffic  Level,  p  =  0.  8 


The  expcetcd  queuing  delay  due  to  buffer¬ 
ing  during  demultiplexing  la  another  Important 
parameter  In  the  dcalgn  consideration  of  statis¬ 
tical  multiplexors,  Sinee  moat  systems  allow  a 
very  low.overflow  probability,  the  expected 
waiting  time  duo  to  buffering  can  be  approxi¬ 
mated  by  thut  of  an  Infinite  wultlng  room  queuing 
model  with  Poisson  arrivals  and  geometric 
service  time.  The  expected  waiting  time  Wj  for 
message  sending  to  the  flh  destination  la 

\E<X.2)  p  (21.-1) 

W  ■  - -  a  ■ 

1  2(1  -pt)  2(1-  pt) 

eharacter-aervlee-tlmea  (5) 

2 

where  E(Xj  )  la  the  aeeond  moment  of  ti.e  mes¬ 
sage  length  Xj  for  the  l*h  destination. 

Since  the  average  arrival  rate  and  the 
average  message  length  for  each  destination  are 
different,  the  expeeted  quoulng  delay  due  to 
demultiplexing  for  oueh  destination  la  also  dif¬ 
ferent  and  should  be  computed  from  Its  associ¬ 
ated  traffic  Intensity  and  average  message 
length. 

HI.  Discussion  of  Results 

The  relationships  between  buffer  size  and 
buffer  overflow  probability  for  selected  average 


■% 
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traffic  levels,  average  burst  lengths,  and  traffic 
destination  functions  are  portrayed  in  Figs.  5 
ami  (i.  For  a  given  average  message  length, 
and  desired  level  of  ov  illow  probability,  the 
required  buffer  size  Increases  ns  the  average 
tr.tlflc  level  Increase  t.  I*  aether,  the  required 
buffer  size  varies  dr.istli  a  11  y  with  the  message 
dcntliiatlmi  linn  tlunii.  (  mi.. p  Hu  ■  I  he  live 

types  of  lr. title  ili  ill  1 1 1<  hi  Mini  I1 . t  1 1:  i i 

In  mu  sJiiiitlalli.il,  till  ,1  ( -  K  .11  .,*.  I  rage  1 1  ,i  I  tie 
level.  Hie  tmlliii  in  <1  •  • . . 1 1 1 .  Mnn  inn.  I  mu  required 
the  smallest  lillll.  i  -,i  aa.l  It.-  lie))  2  llestl- 
n. it luii  function  i  et|iiir.  d  the  i.n.  i  at  Imll'er 
size,  "fills  is  l.ei  ...|.,e  linih  !.  i , • .  r  size  mill 
quelling  delay  1  in  .  ,  ,  .1  .\  |.  .ii.hIi  !  ly  \\  |  tli 
Pj.  Tims,  fora  given  ,n.  .  i,  tu:  level, 

ililferent  type. i  nl  ,1c  .1  h a  ,  t . u  ,  I 
yield  dl I ierent  hill,  i  h.  |.  |..r  |  ,,nple, 

if  one  of  the  d,  1 1 ....  I ,  i  .  i  ,  . .  ,•  1 1 y  loaded,  the 
average  traffic  level  nl  the  it,  m  could  be 
Very  low,  but  an  extri  mely  huge  huffer  Is 
needed.  1  nrlher,  the  .  \p.  i  1 1  . 1  qu<  ulng  tlelay  for 
liiCMilnen  •aiidlnr  h-  Hi.  <  he.  ,|n  h-ailed  desltl- 
IWtloll  would  lie  very  l..i..;.  y,  .  i,  ,.  that  the 

ellei.t  ol  ilieit  1 1 i.d t.n i  lii*  I . .  1. 1, tier  in  ii.ivlul* 

lltcrcanes  as  averag.  ti  1.  •  .  1  Increases. 

Simulation  rennl!.,  m  I.  u  ,  r  behavior 
shed  light  on  the  re  1..I1  ,11  . | | .  1  |„  1 1. 1 'en  the 
dent  jltlpli'XOI  1 1  a  1 1 1  e  1 1 1 1  ai  t  .in  I  the  dcniultl- 

plexlng  system  p.  rloi  . .  .lore  ap,  I  il'ieally, 

the  results  show  that  III  iniia.nl  ••  Ihc  Itl.'.e  of  the 
demultiplexing  built  r  and  Igi.  au.g  ilel.iy  line  to 
buffering,  we  should  acla.dnl  tie  input  .  to  the 
de multiple xor  bnifei*  .. ppi..,ui.,.i.  i  y  equally, 

"I  Ills  can  be  achieved  by  •  -natniiu.g  tlie  volume 
of  input  traffic  to  the  bull,  i  i nr  v..iiotta  destina¬ 
tions,  selecting  the  output  li  an  m.i  i  si  on  r*.  t**  of 
the  buffer,  or  both.  Inn  tim.  ■  *.,h i  mg  . .  • .  t .  1 1 1 , 
Input  truffle  to  the  doimilti,.l.  v.n  Lmil'ei  i  . 
governed  by  the  computer  w la •iiu'iug  algorithm, 
such  as  the  size  of  CPU  uutmUim  tune.  It  we 
know  the  relationship*  between  tin  CPI'  qu  intuin 
time  and  the  volume  of  the  CPU  outputs,  then  u 
variable  quantum  time  scheduling  algorithm 
would  be  effective  to  schedule  an  equal  amount  of 
output  traffic  to  various  destinations.  The  buffer 
output  *rate  can  also  be  changed  to  adjust  the 
traffic  Intensity.  To  achieve  a  uniform  truffle 
destlnation  distribution,  we  should  usalgn  high 
transmission  rate  lines  to  those  destinations 
that  have  high  volumes  of  truffle. 

In  our  simulation  model,  ten  buffer  output 
destinations  are  used.  Since  the  Input  truffle  to 
the  buffer  Is  random,  the  ten  output  destinations 

This  relationship  depends  on  the  computer 
system  and  program  behavior  In  question,  and 
could  be  obtained  via  measurement  or 
simulation. 
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odii  (ilno  lie  viewed  as  ten  groups  of  outputs  pro¬ 
vided  the  outputs  In  each  group  have  approxi¬ 
mately  the  same  value  of  traffic  intensity.  *  When 
a  demultiplexing  buffer  has  more  than  ton  outputs, 
wc  could  group  them  into  ten  groups  according 
to  their  traffic  intensities,  and  then  select  the 
appropriate  graphs  to  estimate  its  buffer 
behavior. 

The  simulation  was  performed  by  using 
the  G I’SS  program  on  the  IflM  300/01  at  UCLA. 
The  computing  time  required  for  the  simulation 
depends  largely  on  the  sample  size  and  the  num¬ 
ber  of  destinations  used.  For  a  given  or, /3,  and  a 
destination  distribution,  the  computing  time 
required  for  an  experiment  (which  represent^  a 
curve  on  the  graph)  of  10'’  samples  and  ten 
destinations  is  about  five  minutes.  As  is  com¬ 
mon  in  many  stochastic  experiments,  their 
results  are  sensitive  with  sample  sizes.  A 
larger  .sample  :u  re  yields  a  more  accurate  esti¬ 
mation.  I''be  iv  compromise  between  accuracy 
and  compel  mg  time  required,  a  sample  size  of 
10  ’  was  selected  lor  each  experiment. 

IV.  I ;  x.implc 

Consider  tie  ijbmign  of  the  demultiplexing 
system  for  a  tin  ■  •  haring  system  that  employs 
the  statistical  multiplexing  technique  as  shown 
in  Fig.  In.  The  input  ii&fflc  to  the  demulti¬ 
plexor  is  i'einT'1.  d  remit  . . nmputer  and  run 

he  approximat'  d  i  |:|ds  him  arrivals.  The 
message  h  ngtli  >  m  he  approximated'  ns  geomet¬ 
rically  dl.strlluil  d  with  a  menu  of  20  characters. 
Further,  the  me  ;sd,;n  dentin, illori  can  be  parti¬ 
tioned  into  ten  I'lmips  oid  ftio  destination  func¬ 
tion  has  a  step  li  e  I .  'Inn  hi  shown  in  Fig.  4c. 

The  average  tr.'ftn  >•  v-  I  I  <  ;  tend  0.  H.  From 
Fig.  (ii),  to  aclllev  e  ,o  •  n  e  j' '  I,  H'-  pi  olvddllLy  of 
10  the  ic(|itlie,l  '•■iii,  •  ■ :  |  •  •  |n  -I,  ilat)  cliarnc- 
ters.  From  Fq.  (  •).  lie-  ex,.,  .  vd  quelling 
delay  due  to  lmin  i  in,'  t  ie  Hu  higher  traffic  des¬ 
tination  group  1  ■>  l.l  ■  Inr.irli  r- 'lei  vlce-tlme.q 
and  tin-  Inwer  ti  it>  :  1 1 1  1 1  i«  m  group  Is  T'i 

ciiararter  — set  »n  •  !•  i  •  i,  I'lirnforc,  the  aver¬ 
age  -sen  b  e  •  I  lee  •  I  <  (  |t  l.TI  clinene- 

ters-scr vie c'-llie  r.n'.v  suppose  we  rhange 

the  CPU  scheihdin:  ui  a  variable  quantum  time 
scheduling  nlgorlilm  .  Ily  a  < .signing  larger 
quantum  time  i  in  Me.  ••  lower  traffic  intensity 
destinations,  the  <1,  Oiii.dimi  function  changes 
from  a  step-id  «•  fnii'Mcn  to  an  approximately 
uniform  functl  'n.  I  rum  Fig.  fill,  the  required 
buffer  size  to  achieve  l],f  1 0 “5  is  2,  200  cha r- 
aeter.s,  and  from  Fq.  (a),  the  queuing  delay  is 
70  character  -nervier  times.  We  note  that  both 
the  required  buffer  size  mid  the  queuing  delay 
duo  to  buffering  have  been  improved  after 

Simulation  results  verified  this  eonjeeture  for 
two  or  three  outputs  In  a  group. 
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changing  the  CPU  scheduling  algoritlim.  In  a 
practical  system  design,  wt  should  further 
consider  the  inquiry  response  time  constraints, 
the  overhead  cost  of  various  scheduling  algo¬ 
rithms,  and  the  CPU  throughput. 

V,  Conclusions 

The  statistical  demultiplexor  consists  of  a 
buffjr  and  a  switching  circuit.  Due  to  the  com¬ 
plex  output  structure  of  the  demultiplexing 
buffer.  Its  behavior  is  studied  via  computer  sim¬ 
ulation.  Simulation  results  reveal  that  the 
traffic  destination  function  has  a  drastic  effect 
on  the  demultiplexing  buffer  behavior.  The  best 
buffer  behavior  is  achieved  by  the  uniform  desti¬ 
nation  function;  that  is,  the  amount  of  traffic 
sending  to  various  destinations  are  equal.  Since  ‘ 
the  CPU  scheduling  algorithm  in  a  time-sharing 
system  strongly  affects  the  traffic  destination 
function,  computer  operating  systems  have  large 
influences  on  demultiplexor  performance.  For 
example,  a  variable  CPU  quantum  time  sched¬ 
uling  algorithm  may  be  effective  in  producing  a 
uniform  destination  function  and  may  yield  better 
demultiplexing  performance.  Thus  the  simula¬ 
tion  model  and  the  results  in  this  paper  serve  as 
a  useful  guide  in  designing  the  demultiplexing 
system  and  in  planning  an  optimal  computer 
communication  system. 
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ABSTRACT 


The  design  of  the  ARPA  Computer  Network  brought 
together  manv  individuals  with  diverse  backgrounds  and 
philosophies.  In  this  paper,  we  review  the  methods  used  in 
the  design  of  the  Network  from  the  vantaoe  of  over  two  years 
experience  in  its  development.  The  design  variables,  system 
constraints,  and  performance  criteria  for  the  network  are 
discussed  along  with  an  evaluation  of  the  tools  used  to 
design  an  efficient  and  reliable  system.  The  design 
procedures  and  the  conclusions  reached  about  the  network's 
properties  appear  to  be  generally  applicable  to  message 
switched  networks.  Conseauently ,  the  results  of  this  paper 
should  be  useful  in  the  design  and  study  of  other 
store-and- forward  computer  communication  networks. 
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INTRODUCTION 


I. 


The  AREA  Network  (ARPANET)  project  brouqht  together 
many  individuals  with  diverse  backqrounds ,  philosophies,  and 
technical  approaches  from  the  fields  of  computer  science, 
communication  theory,  operations  research  and  others.  The 
project  was  aimed  at  providinq  an  efficient  and  reliable 
computer  communications  system  (usinq  messaqe  switchinn 
techniques)  in  which  computer  resources  such  as  proorams , 
data,  storaqe,  special  purpose  hardware  etc.  could  be 
shared  amonq  computers  and  amonq  many  users  £38].  The 
variety  of  desiqn  methods,  ranqinq  from  theoretical  mouelinq 
to  hardware  development,  were  primarilv  employed 
independents,  althouqh  cooperative  efforts  amonq  desinners 
occurred  on  occasion.  As  of  November  1971,  the  network  has 
been  an  operational  facility  for  manv  months,  with  about  20 
participatinq  sites,  a  network  information  center  accessible 
via  the  net,  and  well  over  a  hundred  researchers,  system 
proqrammers,  computer  center  directors  and  other  technical 
and  administrative  personnel  involved  in  its  operation. 

In  this  paper,  we  review  and  evaluate  the  methods  used 
in  the  ARPANET  desiqn  from  the  vantaqe  of  over  two  years 
experience  in  the  development  of  the  network.  In  writinu 
tiiis  paper,  the  authors  have  each  made  equal  contributions 
durinq  a  series  of  intensive  discussions  and  debates. 
Rather  than  present  merely  a  summary  of  the  procedures  that 
were  used  in  the  network  desiqn,  we  have  attempted  to 
evaluate  one  another??  methods  to  determine  their  advantaqos 
and  drawbacks.  Our  approaches  and  philosophies  have  often 
differed  radically  and,  as  a  result,  this  has  not  been  an 
easy  or  undisturbinq  process.  On  the  other  hand,  we  have 
found  our  collaboration  to  be  extrememlv  rewardino  ana, 
remarkably,  we  have  uncovered  many  similar  properties  about 
the  network's  behavior  that  seem  to  be  qenerally  applicable 
to  messaqe  switched  networks. 

The  essence  of  a  network  is  its  desiqn  philosoohv,  its 
performance  characteristics,  and  its  cost  of  implementation 
and  operation.  Unfortunately  there  is  no  qenerally 
accepted  definition  of  an  "optimal"  network  or  even  of  a 
" qood "  network.  For  example,  a  network  deniqned  to  transmit 
larqe  amounts  of  data  onlv  durinq  late  eveninq  hours  mioht 
call  for  structural  and  performance  characteristics  far 
different  from  one  servicinq  large  numbers  of  users  who  are 
rapidly  exchanoinq  short  messaqes  durinq  business  hours.  Ne 
expect  this  topic,  and  others  such  as  the  merits  of  messaqe 
switching  vs.  circuit  switchinq  or  distributed  vs. 
centralized  control  to  be  a  subject  of  discussion  for  manv 
years  Cl, 14, 32, 34]. 
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A  cost  analysis  performed  in  1967-1968  for  the  ARPA 
Network  indicated  that  the  use  of  messaqe  switching  would 
lead  to  more  economical  communications  and  better  overall 
availability  and  utilization  of  resources  than  other  methods 
(34,367.  In  addition  to  its  impact  on  the  availability  of 
computer  resources,  this  decision  has  generated  widespread 
interest  in  store-and-forward  communications.  In  many 
instances,  the  use  of  store-and-forward  communication 
techniques  can  result  in  greater  flexibility,  higher 
reliability,  significant  technical  advantage,  and 
substantial  economic  savings  over  the  use  of  convential 
common  carrier  offerings.  An  obvious  trend  towarus 
increased  computer  and  communication  interaction  has  bequn. 
In  addition  to  the  ARPANET,  research  in  several  laboratories 
is  underway,  small  experimental  networks  are  being  built, 
and  a  few  examples  of  other  government  and  commercial 
networks  are  already  apparent  (6 , 7 , 31 , 40 , 41 , 47 , 4BJ . 

In  the  ARPANET,  each  time-sharinq  or  batcii  procr.ssinn 
computer,  called  a  Host,  is  connected  to  a  amall  computer 
called  an  Interface  Messaqe  Processor  (IMP?.  The  IMl's, 
which  are  interconnected  by  leased  50  kilobit/second 
circuits,  handle  all  netv/ork  communication  for  their  Hosts. 
To  send  a  message  to  another  Host,  a  Host  precedes  the  text 
of  its  messaae  with  an  address  and  simplv  delivers  it  to  its 
IMP.  The  I’Ts  then  determine  the  route,  proviue  error 
control  ,  and  notify  the  sender  of  its  receipt.  The 
collection  of  Hosts,  Imps,  and  circuits  forms  the  messaqe 
switched  resource  sharing  network.  A  good  description  of 
the  ARPANET,  and  some  early  results  on  protocol  development 
and  modelinq  are  given  in  (3,15j.  Some  experimental 
utilization  of  the  ARPANET  is  described  in  (42  j.  A  more 
recent  evaluation  of  such  networks  and  a  forward  look  is 
given  in  (35,39j. 

The  development  of  the  Netv/ork  involved  four  principal 
activities  s 

(])  The  desiqn  of  the  IMPs  to  act  as  nodal  store-and 
forward  switches, 

(2)  the  topological  desiqn  to  soecifv  the  capacity  and 
location  of  each  communication  circuit  within  the 
network , 

(3)  The  desiqn  of  higher  level  protocols  for  the  use  of 
the  netv/ork  bv  time-sharing,  batcii  processing  and 
other  data  processing  systems,  and 

(4)  System  modelinq  and  measurement  of  netv/ork 
performance. 

Each  of  the  first  three  activities  were  essentially 
performed  independently  of  each  other,  whereas  the  modeling 
effort  partly  affected  the  IMP  design  effort,  and  closely 
interacted  with  the  topological  design  project. 


The  IMPs  were  desiqncd  by  Bolt  Bcranek  and  Newman  Inc. 
(BBN )  and  were  built  to  operate  independent  of  the  exact 
network  connectivity;  the  topolocrical  structure  was 
specified  by  Network  Analysis  Corporation  (NAC)  usinq  models 
of  network  performance  developed  by  NAC  and  by  the 
University  of  California  at  Los  Anqeles  (UCLA) .  The  major 
efforts  in  the  area  of  system  modelinq  were  performed  at 
UCLA  usinq  theoretical  and  simulation  techniques.  Network 
performance  measurements  have  been  conducted  durinq  the 
development  of  the  network  by  BBN  and  by  the  Network 
Measurement  Center  at  UCLA.  To  facilitate  effective  use  of 
the  net,  hiqher  level  (user)  protocols  are  under  development 
by  a  group  of  representatives  of  universities  and  research 
centers.  This  group,  known  as  the  Network  Working  Group, 
has  already  specified  a  Host  to  Host  protocol  and  a  Telnet 
protocol,  and  is  in  the  process  of  completing  other  function 
oriented  protocols  Ca,29],  We  make  no  attempt  to  elaborate 
on  the  Host  protocol  desiqn  problems  in  this  paper. 
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II.  THE  NETWORK  DESIGN  PROBLEM 

A  variety  of  performance  requirements  and  system 
contraints  were  considered  in  the  design  of  the  net. 
Unfortunately,  many  of  the  key  design  objectives  had  to  be 
specified  long  before  the  actual  user  requirements  coulu  be 
known.  Once  the  decision  to  employ  message  switching  was 
made,  and  fifty  kilobit/second  circuits  were  chosen,  the 
critical  design  variables  were  the  network  operating 
procedure  and  the  network  topology;  the  desired  values  of 
throughput,  delay,  reliability  and  cost  were  system 
performance  and  constraint  variables.  Other  constraints 
affected  the  structure  of  the  netwtrk,  but  not  its  overall 
properties,  such  as  those  arising  from  decisions  about  the 
length  of  time  a  message  could  remain  within  the  network, 
the  location  of  IMPs  relative  to  location  of  Hosts,  and  the 
number  of  Hosts  to  be  handled  by  a  single  IMP, 

In  this  section,  we  identify  the  central  issues  related 
to  IMP  design,  topological  design,  a*  *  network  modelinq.  In 
the  remainder  of  the  paper,  wo  describe  the  major  design 
techniques  which  have  evolved. 

2.1  IMP  PROPERTIES 

The  key  issv,;e  in  the  design  of  the  IMPs  was  the 
definition  of  a  relationship  between  the  IMP  subnet  and  the 
Hosts  to  partition  responsibilities  so  that  reliable  and 
efficient  operation  would  be  achieved.  The  decision  was 
made  to  build  an  autonomous  subnet,  independent  (as  much  as 
possible)  of  the  operation  of  any  Host.  For  reliability, 
the  IMPs  were  desianed  to  be  robust  against  all  line 
failures  and  the  vast  majority  of  IMP  and  Host  failures. 
This  decision  required  routing  strategies  that  dynamicallv 
adapt  to  changes  in  the  states  of  IMPs  and  circuits,  an 
elaborate  flow  control  strategy  to  protect  the  subnet 
adainst  Host  malfunction  and  congestion  due  to  IMP  buffer 
limitations.  In  addition,  a  statistics  and  status  reporting 
mechanism  was  needed  to  monitor  the  behavior  of  the  network. 

The  number  of  circuits  that  an  IMP  must  handle  is  a 
desiqn  constraint  directly  affecting  both  the  structures  of 
the  IMP  and  the  topological  design.  The  speed  of  the  IMP 
and  the  required  storage  for  program  and  buffers  depenu 
directly  upon  the  total  required  processing  capacity,  which 
must  be  high  enough  to  switch  traffic  from  one  line  to 
another  when  all  are  fully  occupied.  Of  great  importance  is 
the  property  that  all  IMPs  operate  with  identical  programs. 
Thi*  technique  greatly  simplifies  the  problem  of 
understanding  network  operation  and  makes  network 
modifications  easy  to  perform. 
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The  detailed  physical  structure  of  the  IMP  is  not 
discussed  in  this  paper  t2,15J.  However,  the  operating 
procedure,  which  guides  packets  through  the  net,  is  very 
much  of  interest  here.  The  flow  control,  routing,  and  error 
control  techniques  are  integral  parts  of  the  operating 
procedure  and  can  be  studied  apart  from  the  hardware  by 
which  they  are  implemented.  Most  hardware  modifications 
require  changes  to  many  IMPs  already  installed  in  the  field, 
while  a  change  in  the  operating  procedure  can  often  be  made 
more  conveniently  by  a  change  to  the  single  operating 
program  common  to  all  IMPs,  which  can  then  be  propagated 
from  a  single  location  via  the  net. 

2.2  TOPOLOGICAL  PROPERTIES 

The  topological  design  resulted  in  the  specification  of 
the  location  and  capacity  of  all  circuits  in  the  network. 
Projected  Host  -  Host  traffic  estimates  were  known  at  the 
start  to  be  either  unreliable  or  wrong.  Therefore,  the 
network  was  designed  under  the  assumption  of  equal  traffic 
between  all  pairs  of  nodes.  (Additional  superimposed 
traffic  was  sometimes  included  for  those  nodes  with 
expectation  of  higher  traffic  requirements) .  The 
topological  structure  was  determined  with  the  aid  of 
specially  developed  heuristic  programs  to  achieve  a  low 
cost,  reliable  network  with  a  high  throughput  and  a  general 
insensitivity  to  the  exact  traffic  distribution.  Currently, 
only  50  kilobit/second  circuits  are  being  used  in  the 
ARPANET .  This  speed  line  was  chosen  to  allow  rapid 
transmission  of  short  messages  for  interactive  processing, 
as  well  as  to  achieve  high  throughput  for  transmission  of 
long  messages.  For  reliability,  the  network  was  constrained 
to  have  at  least  two  independent  paths  between  each  pair  of 
IMPs. 


The  topological  design  problem  requires  consideration 
of  the  following  two  questions: 

(1)  Starting  with  a  given  state  of  the  network 
topology,  what  circuit  modifications  are  required  to 
add  or  delete  a  set  of  IMPs? 

(2)  Starting  with  a  given  state  of  network  topology, 
when  and  where  should  circuits  be  added  or  deleted  to 
account  for  long  term  changes  in  network  traffic? 

If  the  locations  of  all  network  nodes  are  known  in 
advance,  it  is  clearly  most  efficient  to  design  the 
topological  structure  as  a  single  global  effort.  However, 
in  the  ARPANET,  as  in  most  actual  networks,  the  initial 
designation  of  node  locations  is  modified  on  numerous 
occasions.  On  each  such  occasion,  the  topology  can  be 
completely  reoptimized  to  determine  a  new  set  of  circuit 
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locations . 

In  practice,  there  is  a  long  lead  time  between  the 
ordering  and  the  delivery  of  a  circuit  and  major  topological 
modifications  cannot  be  made  without  substantial  difficulty. 
It  is  therefore  prudent  to  add  or  delete  nodes  with  as 
little  disturbance  as  possible  to  the  basic  network 
structure  consistent  with  overall  economical  operation. 
Fiqure  1  shows  the  evolution  of  the  ARPANET  from  the  basic 
four  IMP  desiqn  in  1969  to  the  presently  planned  26  IMP 
version.  Inspection  of  the  24  and  26  IMP  network  designs 
reveals  a  few  substantial  changes  in  topology  that  take 
advantage  of  the  new  nodes  being  added.  Surprisingly 
enough,  a  complete  reoptimization  of  the  26  IMP  topology 
yields  a  network  only  slightly  less  expensive  (about  1%  per 
year)  than  the  present  network  design  l28J. 

2.3  NETWORK  MODELS 

The  development  of  an  accurate  mathematical  model  for 
the  evaluation  of  time  delay  in  computer  networks  is  among 
the  more  difficult  of  the  topics  discussed  in  this  Wer. 
On  the  one  hand,  the  model  must  properly  reflect  the 
relevant  features  of  the  network  structure  and  operation, 
including  practical  constraints.  On  the  other  hand,  the 
model  must  result  in  a  mathematical  formulation  which  is 
tractible  and  from  which  meaningful  results  can  be 
extracted.  However,  the  two  requirements  are  often 
incompatible  and  we  search  for  an  acceptable  compromise 
between  these  two  extremes . 

The  major  modeling  effort  thus  far  has  been  the  study 
of  the  behavior  of  networks  of  queues  £1213 .  This  emphasis 
is  logical  since  in  message  switched  systems,  messages 
experience  queueing  delays  as  they  pass  *ro*  "od«  *°nod* 
and  thus  a  significant  performance  measure  is  the  speed  at 
which  messages  can  be  delivered.  The  c?ueu®1^^od^!w^ 
developed  at  a  time  when  there  were  no  operational  networks 
available  for  experimentation  and  model  va^dation,and 
simulation  was  the  only  tool  capable  of  testing  tl 
validity.  The  models,  which  at  all  times  were  recognized  to 
be  idealized  statements  about  the  real  network,  were 
nonetheless  crucial  to  the  ARPANET  topological  design  effort 
since  they  afforded  the  only  known  way  to  quantitatively 
SaSoT  the  properties  of  different  routing  schemes  and 
topological  structures.  The  models  have  been  subsequently 
demonstrated  to  be  very  accurate  predictors  of  network 
throughput  and  indispensable  in  providing  analytical  insight 
into  the  network's  behavior. 

The  real  problem  confronting  the  designer  is  to  create 
a  system  which  provides  suitable  network  performance  at  an 
acceptable  system  cost.  In  particular,  for  given 
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performance  constraints  on  averaqc  packet  delay,  throughput, 
reliability  and  total  network  cost,  he  must  desiqn  the  IMP 
hardware  and  its  operatinq  procedure  ,  select  the  topoloqy, 
and  assign  capacities  to  each  channel.  Tnis  desiqn  must 
typically  be  conducted  without  the  benefit  of  accurate  (or 
often  even  reasonable)  estimates  of  the  user  traffic 
requirements.  Futhermore,  the  design  is  constrained,  for 
example,  by  the  discrete  nature  of  the  available  channel 
capacities,  and  the  number  of  channels  which  an  IMP  can 
service.  A  design  constraint  for  the  ARPANET  is  that,  on 
the  average,  a  packet  must  travel  from  origin  to  destination 
in  0.2  seconds.  This  complete  problem  formulation  is 
sufficiently  complex  that  it  is  difficult  to  beqin  the 
mathematical  analysis,  much  less  to  obtain  an  "optimum" 
solution.  The  key  to  the  successful  development  of 
tractible  models  has  been  to  factor  the  problem  into  a  set 
of  simpler  queueinq  problems.  There  are  also  heuristic 
design  procedure  that  one  can  use  in  this  case.  These 
procedures  seem  to  work  quite  well  and  are  described  later 
in  the  paper. 

However,  if  one  soecializes  the  problem  and  removes 
some  of  the  real  constraints,  theory  and  analysis  become 
useful  to  provide  understanding,  intuition  and  desiqn 
guidelines  for  the  original  constrained  problem.  This 
approach  uncovers  global  properties  of  network  behavior, 
which  provide  keys  to  good  heuristic  design  procedures  and 
ideal  performance  bounds. 
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in.  DESIGN  TECHNIQUES 

In  this  section  we  describe  the  approaches  taken  to  the 
desiqn  problems  introduced  in  Section  II.  We  first 
summarize  the  important  properties  of  the  ARPANET  design: 


(1) 

(2) 

(3) 


A  communications  cost  of  less  than  30  cents  per 
thousand  packets (approximately  a  megabit). 

Average  packet  delays  under  0.2  seconds  through  the 

ne  ^ 

Capacity  for  expansion  to  64  IMPS  without  major 
hardware  or  software  redesign.  . „ 

(4)  Average  total  throughput  capability  of  10  -  lb 
kilobits/second  for  all  Hosts  at  an  I. IP. 

(5)  Peak  throughput  capability  of  85  kilobits/second 
per  pair  of  IMPS  in  an  otherwise  unloaded  network. 

(6)  Transparent  communications  with  maximum  message 
size  of  approximately  8000  bits  and  error  rates  of 
one  bit  in  lo'*’  or  less. 

(7)  Approximately  98%  availability  of  any  IMP  and  close 
to  100%  availability  of  all  operating  IMPs  from  anv 
operable  IMP. 

The  relationships  between  the  various  design  efforts  is 
illustrated  by  these  properties.  The  t°f,°1°^ca^eS^o 
nrovides  for  both  a  desired  average  throughout  and  for  two 
or  more  paths  to  be  fullv  used  for  communication  between  anv 
pair  of  Hosts. The  operating  procedure  should  allow  any  pair 
of  Hosts  to  achieve  those  objectives.  Tne  avaiiability^ 
IMPs  to  communicate  reflects  both  the  fact  1  ,  ,  ^ 

down  about  2%  of  the  time  and  that  the  topology  is  selectee 
so  that  circuit  failures  contribute  little  additional  to  the 
total  system  downtime. 

3.1  IMP  DESIGN 

The  IMP  design  consists  of  two  closely  coupled  but 
nonetheless  separable  pieces  -  the  P^-cal  hardware 
snecification  (based  on  timing  and  reliability 
considerations  and  the  operating  procedure)  and  the  design 
and  implementation  of  the  operating  procedure  using  the 
specified  IMP  hard'^are.  The  IMP  originally  developed  for 
the  ARPANET  contains  a  16-bit  one  microsecond  computer  that 
can  handle  a  total  bandwidth  of  approximately  on 

meqabit/second  (e.g.  twenty  50  ihe» 

all  the  messages  are  maximum  size.  In  the  worst  case  w 
all  the  messages  are  short,  the  IMP  capacity  is  about  0.3 
meaabits /second  C15J.  Hardware  is  likely  to  change  as 
functionof  the  require  IMP  capacity  but  an  operating 
procedure  that  operates  well  at  one  IMP  capacity  is  likely 
to  be  transferable  to  machines  that  provide  differen 
capacity.  However,  as  a  network  grows  in  size  and 
utilization,  a  more  comprehensive  operating  procedure 
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takes  account  of  known  structural  properties,  such  as  a 
hierarchical  topology,  may  be  appropriate. 

Four  primary  areas  of  IMP  design,  namely  message 
handling  and  buffering,  error  control,  flow  control,  and 
routing  are  discussed  in  this  section.  The  IMP  provides 
buffering  to  handle  messages  for  its  Host  and  packets  for 
other  imps.  Error  control  is  requireu  to  provide  reliable 
communication  of  Host  messages  in  the  presence  of  noisy 
communication  circuits. 

The  design  of  the  operating  procedure  should  allow  high 
throughput  in  the  net  under  heavy  traffic  loads.  Two 
potential  obstacles  to  achieving  this  objective  are:  (1)  The 
net  can  become  congested  and  cause  the  throughput  to 
decrease  with  increasing  load,  and  (2)  The  routing  procedure 
may  be”unable  to  always  adapt  sufficiently  fast  to  the  rapid 
movement  of  packets  to  insure  efficient  routinq.  A  Flow 
Control  and  Routing  procedure  is  needed  that  can  efficiently 
meet  this  requirement. 


3.1.1  Message  Handling  and  Buffering 

In  the  ARPANET,  the  maximum  messaue  size  was 
constrained  to  be  approximately  8000  bits.  A  pair  of  hosts 
will  typically  communicate  over  the  net  via  a  sequence  of 
transmitted  messages.  To  obtain  delays  of  a  few  tenths  of  a 
second  for  such  messages  and  to  lower  the  requireu  IMP 
buffer  storage,  the  IMP  program  partitions  each  message  into 
one  or  more  packets  each  containing  at  most  approximately 
1000  bits.  Each  packet  of  a  message  is  transmitted 
independently  to  the  destination  where  the  message  is 
reassembled  by  the  IMP  before  shipment  to  that  destination 
Host.  Alternately,  the  Hosts  could  assume  the 
responsibility  for  reassembling  messaqes.  For  an 
asynchronous  IMP-Host  channel,  this  sliqhtlv  simplifies  the 
IMP'S  task.  However,  if  ever v  IMP-Host  channel  is 
synchronous,  and  the  Host  provides  the  reassembly,  the  I  U 
task  is  considerably  simplified  at  the  expense  of  I.tP-like 
softv/are  in  each  Host  and  error  control  on  the  I.  1  -Host 
circuit.  Since  each  Host  would  require  a  different  program 
to  perform  these  functions,  this  would  be  a  very  costlv 
process  compared  to  the  simplicity  of  writing  a  common 
program  for  all  IMPs.  On  balance,  it  is  least  costlv 
provide  reassembly  by  the  IMPs  or  to  restrict  the  message 
lengths  to  a  single  packet  at  the  possible  expense  of 
lowered  throughput. 

The  method  of  handlinq  buffers  should  be  simple  to 
allow  for  fast  processing  and  a  small  amount  of  program. 
The  number  of  buffers  should  be  sufficient  to  store  enoucj.i 
packets  for  the  circuits  to  be  used  to  capacity;  the  size  °_ 
the  buffers  may  be  intuitively  selected  with  the  aid  o 
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For  example, 

simple  analytical  teC .l^pful^in  the  IMP  for  simplicity  of 
fixed  buffer  sizes  are  '  Qn  but  inefficient  utilization 
desiqn  and  speed  of  P  .  ,  '  ienqth  packets.  If  each 
can  arise  because  of  ’ ^  overhead  and  provides  space  for  M 
buffer  contains  A  word  meSsaqe  sizes  are  uniformly 

words  of  text,  and  can  be  sbown  f4  5)  that  tne 

distributed  between  1  and  ,  expected  storaqe  -s 

choice  of  M  t5Sf  chosen  to  be 

approximately  7^'  arc„Ltion  that  most  traffic  will  be 
somewhat  smaller  °lh^° a^untP of  overhead  can  be  as  much  as, 

say^  25%  of  buffer  storaqe. 

3.1.2  Error  Control 

...o.’SSnSK  .“S.S8“  to°consibcr^lnn 

„essa,es  are  delivered  to  their  destination  out  of 

(2,  Duplicate  nossaoes  are  delivered  to  the 

destination.  orrnre 

(3)  Messaqes  arc  delivered  with  errors. 

(4)  Messages  are  not  deliverer  . 

c  frtprr Afjcs  for  delivcrv 

The  task  of  P-P-^-^^SaUv  faUs1n°the  province  of 
to  the  destination  Host  actual  Y  onc  nGssaac: 

both  error  control  and  flow  “f^J-1)e”Cen  a  pair  of  Hosts 
at  a  tine  is  ‘“"“d  A  dusUatu  PKk«t  »iU 

proper  sequencinq  occurs  natur  -  knov,iedqnent  has 

arrive  at  the  ««»““ “f  .^Mly  received  packet  to 
been  missed,  thus  causi  q  can  handle  the  first  two 

be  retransmitted.  ilie  upnce  number  to  each  packet  as 

conditions  by  assiqninq  a-  ssinq  the  seauencc  number  a 

it  enters  the  network  ana  p  performs  reassembly  can 

the  destination  IMP.  A  Host seauence  numbers  and  check  for 
also  assiqn  and  process  seq^^cations ,  the  order  of 
dupl icate  packets.  /«  ^“ate^l.  For  priority 

delivery  to  the  destination  is^^  ^  cMC  that  fast 

delivery '  requires '  a  ^rturbation  to  the  sequence. 

•  i  .  pa11oPd  by  noise  on  tne 
Errors  are  Pr^ar^X  rp  handled  most  simply  by  error 
communication  circuits  and  are  ha  each  pair  of  IMPS  alona 
detection  and  retransmission^etween^^^^  requir£JS  extra 

the  transmission  ?ath* . . h  r  circuit  speeds  or  circuit 
storaqe  in  the  IMP  if  increase.  Failures  in  detectina 
lenqths  substantially  '  tbe  order  of  years  to 

errors  can  he  »ade  to  ocour^  °"tr‘hve?taad  (20  -30  parity 
centuries  apart  kilobit/second  circuits  1 

bits  per  packet  with  the  '  detection  codes  may  be 

ARPANET).  Standard  cycli 
usefully  applied  here. 
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A  reliable  system  design  insures  that  each  transmitted 
message  is  accurately  delivered  to  its  intended  destination. 
The  occasional  time  when  an  IMP  fails  and  destroys  a  useful 
in-transit  message  is  likely  to  occur  far  less  often  than  a 
similar  failure  in  the  Hosts  and  has  proven  to  be 
unimportant  in  practice,  as  are  errors  due  to  IMP  memory 
failures.  A  simple  end  to  end  retransmission  strategy  will 
protect  against  these  situations,  if  the  practical  need 
should  arise.  However,  the  IMPs  must  be  designed  so  that 
they  can  be  removed  from  the  network  without  destroying 
their  internally  stored  packets. 

3.1.3  Flow  Control 

A  network  in  which  packets  may  freely  enter  and  leave 
can  become  congested  or  logically  deadlocked  and  cause  the 
movement  of  traffic  to  halt  £5,17).  Flow  control  techniques 
are  required  to  prevent  these  conditions  from  occurring. 
The  provision  of  extra  buffer  storage  will  mitigate  against 
congestion  and  deadlocks,  but  cannot  in  general  prevent 
them. 


The  sustained  failure  of  a  destination  Host  to  accept 
packets  from  its  IMP  at  the  rate  of  arrival  will  cause  the 
net  to  fill  up  and  become  congested.  Two  kinds  of  logical 
deadlocks,  known  as  reassembly  lockup  and  store-nnd- forward 
lockup  may  occur.  In  reassembly  lockup,  packets  of 
partially  reassembled  messages  are  blocked  from  reaching  tne 
destination  IMP,  (thus  preventing  the  message  from  bcino 
completed  and  the  reassembly  space  freed) ,  by  other  packets 
in  the  net  that  are  waiting  for  reassembly  space  at  that 
destination  to  become  free.  In  a  store  and  forward  lockup, 
the  destination  has  room  to  accept  arriving  packets,  but  the 
packets  interfere  with  each  other  by  tying  up  buffers  in 
transit  in  such  a  way  that  none  of  the  packets  are  able  to 
reach  the  destination  £17).  These  phenomena  have  only  been 
made  to  occur  during  very  carefully  arranged  testina  of  the 
ARPANET  and  by  simulation  £49 J. 

In  the  original  ARPANET  design,  the  use  of  software 
links  and  RFf.’MS  protected  against  conaestion  by  a  single 
link  or  a  small  set  of  links.  However,  the  combined  traffic 
on  a  large  number  of  links  could  still  produce  connestion. 
Although  this  strategy  did  not  protect  aaainst  lockup,  for 
the  levels  of  traffic  encountered  by  the  net  to  date,  the 
method  has  provided  ample  protection. 

A  particularly  simple  flow  control  alqorithm  that 
augments  the  original  IMP  design  to  prevent  congestion  and 
lockup  is  also  described  in  £17).  This  scheme  includes  a 
mechanism  whereby  packets  may  be  discarded  from  the  net  at 
the  destination  IMP  when  congestion  is  about  to  occur,  with 
a  copy  of  each  discarded  packet  to  be  retransmitted  a  short 
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time  later  by  the  originating  Host's  IMP.  Rather  than 
experience  excessive  delays  within  the  net  as  traffic  level 

the  ^afflC  iS  queued  outside  the  net  so  tha 

the  transit  cime  delays  internal  to  the  net  continue  to 

trafficSifto‘thThl£+-S^ate?y  prevents  the  insertion  of  more 
traffic  into  the  net  than  it  can  handle. 


do l  aiic  ^  lmpo^tant  note  the  dual  requirement  for  small 
fast^tran?fp  interactive  traffic  and  high  bandwidth  for  the 
f  of  files.  To  allow  high  bandwidth  between  a 

LJZL'  *  '  net  must  be  able  to  accept  a  steady  flow 

of  packets  from  one  Host  and  at  the  same  time  be  able  to 

tho1olYn?UefC^  the  fl°W  at  thG  entrance  to  the  source  IMP  in 
the  event  of  imminent  congestion  at  the  destination.  This 

alaoHfhmre?UireS  4.thffc  u  separate  Provision  be  made  in  the 
<  Jth •  1  Protect  short  interactive  messages  from 
experiencing  unnecessarily  high  delays. 

3.1.4  Routing 

detwork  routing  strategies  for  distributed  networks 
require  routing  decisions  to  be  made  with  only  information 

toaieff^rft0*-hn  LMP  and  rhe  IfIP  musfc  GXocutG  those  decisions 
to  effect  the  routing  Cl4,15J.  A  simple  example  of  such  a 

strategy  is  to  have  each  IMP  handling  a  packet  independently 
route  it  along  its  current  estimate  of  the  shortest  path  to 
the  destination.  ^ 

For  many  applications,  it  suffices  to  deal  with  an 
i  ealized  routinq  stratecrv  which  may  not  simulate  the  IMP 
routing  functions  in  detail  or  which  uses  information  not 
availabie  to  the  IMP.  The  general  properties  of  both 
strategies  are  usually  similar  ,  differing  mainly  in  certain 
implementation  details  such  as  the  availability  of  buffers 
or  the  constraint  of  counters  and  the  need  for  the  routing 
to  quickly  adapt  to  changes  in  IMF  and  circuit  status. 

The  IMPS  perform  the  routinq  computations  using 
information  received  from  other  IMPs  and  local  information 
such  as  the  alive/dead  state  of  its  circuits.  In  the  normal 
case  of  tine  varying  loads,  local  information  alone,  such  as 
the  length  of  internal  queues,  is  insufficient  to  provide  an 
efficient  routing  strategy  without  assistance  from  the 
neighboring  IMPs.  It  is  possible  to  obtain  sufficient 
information  from  the  neighbors  to  provide  efficient  routing, 
with  a  small  amount  of  computation  needea  per  IMP  and 
without  each  IMP  requiring  a  topological  map  of  the  network 
applications  where  traffic  patterns  exhibit 
the  use  of  a  central  controller  might  even  be 
However,  for  most  applications  which  involve 
varying  traffic  flow,  it  appears  that  a  central 
controller  cannot  be  used  more  effectively  than  the  IMPs  to 
update  routing  tables  if  such  a  controller  is  constrained  to 


In  certain 
regularity, 
preferable . 
dynamically 


n  ui 


Page  ]  4 


Sl^Vf,itS  in,ormation  via  the  net.  It  is  also  a  less 
reliable  approach  to  routing  than  to  distribute  the  routing 
decisions  among  the  iriPs.  routine 

net  ?nG  Q,°^nCT  i"fof!iation  cannot  be  propagated  about  the 
net  in  sufficient  time  to  accurately  characterize  the 
instantaneous  traffic  flow.  An  efficinn  ;tS 

jackets rG' but  r^h  ^  ^  °n  the  m°VGmCnt  °f  individual 

packets,  but  rather  use  topological  or  statistical 

information  in  the  selection  of  routes.  For  exanJlo t 

madeg  procct!ure'  the  flow  of  traffic  can'  be 

™?dc  bu*ld  up  smoothly.  This  allows  the  routine 

algorithm  ample  time  to  adjust  its  tables  in  each  I.’T  in 
advance  of  the  buildup  of  traffic. 

imp  S?«J?hene  oric?lnally  used  in  the  ARPA  network  had  each 
IhP  select  one  output  line  per  destination  onto  which  to 
route  packets.  The  line  was  chosen  to  be  the  one  wiJh 

minimum  estimated  time  delay  to  the  destination.  The 
selection  was  updated  every  half  second  using  minimum  time 

!  i!?at;8,  fr°n  the  neidhb°ring  IMPS  and  internal  estimates 

of  the  delay  to  each  of  the  neiahbors.  tven  thouaS  the 

routine  algorithm  only  selects  one  line  at  a  tine  nor 
destination,  two  output  lines  will  be  used  if  a  queue  PQf 
packets  waiting  transmission  on  one  line  builds  un  before 
the  routing  update  occurs  and  another  line  is  chosen 
Modifications  to  the  scheme  which  allow  several  lines  per 
destination  to  be  used  in  an  update  interval  (during  which 
the  routing  is  not  changed)  are  possible  using  two  or  more 
time  delay  estimates  to  select  the  paths. 

.  In,  Practice,  this  approach  has  worked  quite  effectively 
the  ™odf£?te  levels  of  traffic  experienced  in  the  net. 
For  heavy  traffic  flow,  this  strategy  will  bo  inefficient 
since  the  routing  information  is  based  on  the  length  of 
queues,  which  we  have  seen  can  change  much  faster  than  the 
information  about  the  change  can  be  distributed 

Fortunately,  this  information  is  still  usable,  although  it 
Gar\  w  substantially  out  of  date  and  will  not,  in  general, 
be  helpful  in  making  efficient  routing  decisions  in  tno 
heavy  traffic  case. 


A  more  intricate  scheme,  recently  developed  by  BBd 
allows  multiple  paths  to  be  efficiently  used  even  during 
heavy  traffic  U8J.  Preliminary  simulation  studies  indicate 
that  it  can  be  tailored  to  provide  efficient  routing  in  a 

nC5l!°5k  Wlth  3  variet,/  of  heavy  traffic  conditions.  '  This 
method  separates  the  problem  of  defining  routes  onto  which 
packets  may  be  routed  from  the  problem  of  selecting  a  route 
when  a  particular  packet  must  be  routed.  By  this  technique. 
TMr.1S  P°®slbl°  ho  send  packets  down  a  path  with  the  fewest 
I.IPs  and  excess  capacity  ,  or  when  that  path  is  filled,  the 
one  with  the  next  fewest  IMPs  and  excess  capacity,  etc. 
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A  similar  approach  to  routing  was  independently  derived 
by  NAC  using  an  idealized  method  that  did  not  require  the 
IMPs  to  participate  in  the  routing  decisions.  Another 
approach  using  a  flow  deviation  technique  has  recently  been 
studied  £13 J.  The  intricacies  of  the  exact  approach  lead  to 
a  metering  procedure  that  allows  the  overall  network  flow  to 
be  changed  slowly  for  stability  and  to  perturb  existing  flow 
patterns  to  obtain  an  increased  flow.  These  approaches  all 
possess,  in  common,  essential  ingredients  of  a  desirable 
routing  strategy. 

3.2  TOPOLOGICAL  CONSIDERATIONS 

An  efficient  topological  design  provides  a  high 
throughput  for  a  given  cost.  Although  many  measures  of 
throughput  are  possible,  a  convenient  one  is  the  average 
amount  of  traffic  that  a  sinqle  IMP  can  send  into  the 
network  when  all  other  IMPs  are  transmitting  according  to  a 
specified  traffic  pattern.  Often,  it  is  assumed  that  all 
other  IMPs  are  behaving  identically  and  each  IMP  is  sending 
equal  amounts  of  traffic  to  each  other  IMP.  The  constraints 
on  the  topological  design  are  the  available  common  carrier 
circuits,  the  target  cost  or  throughput,  the  desired 
reliability,  and  the  cost  of  computation  required  to  perform 
the  topological  design. 

Since,  there  was  no  clear  specification  of  the  amount 
of  traffic  that  the  network  would  have  to  accomodate 
initially,  it  was  first  constructed  with  enough  excess 
capacity  to  accomodate  anv  reasonable  traffic  requirements. 
Then  as  new  IMPs  were  added  to  the  system,  the  capacity  was 
and  is  still  being  systematically  reduced  until  the  traffic 
level  occupies  a  substantial  fraction  of  the  network's  total 
capacity.  At  this  point,  the  net's  capacity  will  be 
increased  to  maintain  the  desired  percentage  of  loading.  At 
the  initial  stages  of  network  design,  the  " two-connected" 
reliability  constraint  essentially  determined  a  minimum 
value  of  maximum  throughput.  This  constraint  forces  the 
average  throughput  to  be  in  the  ranqe  10-15  kilobits  per 
second  per  IMP  since  two  communication  paths  between  every 
pair  of  IMPs  are  needed.  Alternatively,  if  this  level  of 
throughput  is  required,  then  the  reliability  specification 
of  "two-connectivity"  can  be  obtained  without  additional 
cost. 

3.2.]  Reliability  Computations 

A  simple  and  natural  characterization  of  network 
reliability  is  the  ability  of  the  network  to  sustain 
communication  between  all  operable  pairs  of  IMPs.  For 
design  purposes,  the  requirement  of  two  independent  paths 
between  nodes  insures  that  at  least  two  IM.Ps  and/or  circuits 
must  fail  before  any  pair  of  operable  IMPs  cannot 
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communicate.  This  criterion  is  independent  of  the 
properties  of  the  IMPs  and  circuits,  does  not  take  into 
account  the  "degree"  of  disruption  that  may  occur  and  hence, 
does  not  reflect  the  actual  availability  of  resources  in  the 
network.  A  more  meaninqful  measure  is  the  average  fraction 
of  IMP  pairs  that  cannot  communicate  because  of  IMP  and 
circuit  failures.  This  calculation  requires  knowledge  of 
the  IMP  and  circuit  failure  rates,  and  could  not  be 
performed  until  enough  operating  data  was  gathered  to  make 
valid  predictions. 

To  calculate  network  reliability,  we  must  consider 
elementary  network  structures  known  as  cutsets.  A  cutset  is 
a  set  of  circuits  and/or  IMPs  whose  removal  from  the  network 
breaks  all  communication  paths  between  at  least  two  operable 
IMPs.  To  calculate  reliability,  it  is  often  the  case  that 
all  cutsets  must  be  either  enumerated  or  estimated.  As  an 
example  ,  in  a  23  IMP,  28  circuit  ARPA  Network  design 
similar  to  the  one  shown  in  Figure  1(d),  there  are  over 
twenty  million  ways  of  deleting  only  circuits  so  that  the 
remaining  network  has  at  least  one  operable  pair  of  IMPs 
with  no  intact  communication  paths.  Table  1  indicates  the 
numbers  of  cutsets  in  the  23  IMP  network  as  a  function  of 
the  number  of  circuits  they  contain. 

A  combination  of  analvsis  and  simulation  can  be  used  to 
comnute  the  average  fraction  of  non-communicating  IMP  pairs. 
Detailed  descriptions  of  the  analysis  methods  are  given  in 
C44J  while  their  application  to  the  analysis  of  the  ARPANLT 
is  discussedin  £43J.  The  results  of  an  analysis  of  the  23 
IMP  version  of  the  network  are  shov/n  in  Ficrure  2.  The  curve 
marked  A  shows  the  results  under  the  assumption  that  IMPs  do 
not  fail,  while  the  curve  marked  B  shows  the  case  where 
circuits  do  not  fail.  The  curve  marked  C  assumes  that  both 
IMPs  and  circuits  fail  with  equal  probability.  In  actual 
operation,  the  average  failure  probability  of  both  IMPs  and 
circuits  is  about  C.02.  For  this  value,  it  can  be  seen  that 
the  effect  of  circ.  ' _  failures  is  far  less  significant  than 
the  effect  of  IMP  failures.  If  an  IMP  fails  in  a  network 
with  n  IMPs,  at  least  n-1  other  IMPs  cannot  communicate  with 
it.  Thus,  good  network  design  cannot  improve  upon  the 
effect  directly  due  to  IMP  failures,  which  in  the  ARPANET  is 
the  major  factor  affecting  rcliabilty.  Further,  more 
intricate  reliability  analyses  which  consider  the  loss  of 
throughout  capacity  because  of  circuit  failures  have  also 
been  performed  and  these  losses  shown  to  be  negligible  C28}. 
Finallv,  unequal  failure  rates  due  to  differences  in  line 
lengths  have  been  shown  to  have  only  minor  effects  on  the 
analysis  and  can  usually  be  neglected  I27J. 


3.2.2  Topological  Optimization 


During  the  computer  optimization  process,  the 
reliability  of  the  topology  is  assumed  to  be  acceptable  if 
the  network  is  at  least  two-connected.  The  object  of  the 
optimization  is  to  decrease  the  ratio  of  cost  to  throughout 
subject  to  an  overall  cost  limitation.  This  technique 
employs  a  sophisticated  network  optimization  program  that 
utilizes  circuit  exchange  heuristics,  routing  and  flow 
analysis  algorithms,  to  generate  low  cost  designs.  In 
addition,  two  time  do lav  models  were  initiallv  useu  to ( 
1) calculate  the  throughput  corresponding  to  to  an  averano 
tine  delay  of  0.2  seconds,  (2)  estimate  the  packet  rejection 
rate  due  to  all  buffers  filling  at  an  IMP.  As  experience 
with  these  models  grew,  the  packet  rejection  rate  was  founa 
to  be  negligible  and  the  computation  discontinued.  The 
delay  computation  (equation  (7)  in  section  3.3.2)  was 
subsequentlv  first  replaced  by  a  heuristic  calculation  to 
speed  the  computation  and  later  eliminated  after  it  was 
found  that  time  delays  could  be  guaranteed  to  be  acceptably 
low  by  preventing  cutsets  from  beinq  saturated.  Thin 
"threshold"  behavior  in  discussed  further  in  section  3.3. 

The  basic  method  of  optimization  v/as  described  in  C12) 
while  extensions  to  the  design  of  .largo  networks  are 
discussed  in  C9]  .  The  method  operates  by  initially 
generating,  either  manually  or  by  computer,  a  "starting 
network"  that  satisfies  the  overall  network  constraints  but 
is  not,  in  general,  a  low  cost  network.  The  computer  tr.cn 
iteratively  modifies  the  starting  network  in  simple  .stems 
until  a  lower  cost  network  is  found  that  satisfies  the 
constraints  or  the  process  is  terminated.  The  process  is 
repeated  until  no  further  improvements  can  be  found.  Using 
a  different  starting  network  can  result  in  a  different 
solution.  However,  by  incorporating  sensible  heuristics  and 
by  using  a  variety  of  carefully  chosen  startinq  networks  and 
some  degree  of  man-machine  interaction,  "excellent"  final 
networks  usually  result.  Experience  has  shown  that  there 
are  a  wide  variety  of  such  networks  with  different 
topological  structures  but  similar  cost  and  performance. 

The  kev  to  this  desiqn  effort  is  the  heuristic 
procedure  by  which  the  iterative  network  modifications  are 
made.  The  method  used  in  the  ARPANET  design  involves  the 
removal  and  addition  of  one  or  two  circuits  at  a  time.  Many 
methods  have  been  employed,  at  various  times,  to  identify 
the  appropriate  circuits  for  potential  addition  or  deletion. 
For  example,  to  delete  uneconomical  circuits  a 
straightforward  procedure  simply  deletes  single  circuits  in 
numerical  order,  reroutes  traffic  and  reevaluates  cost  until 
a  decrease  in  cost  per  megabit  is  found.  At  this  point,  the 
deletion  is  made  permanent  and  the  process  begins  again.  A 
somewhat  more  sophisticated  procedure  deletes  circuits  in 
order  of  increasing  utilization,  while  a  more  complex  method 
attempts  to  evaluate  the  effect  of  the  removal  of  any 
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circuit  before  any  deletion  is  attempted.  The  circuit  with 
the  greatest  likelihood  of  an  improvement  is  then  considered 
for  removal  and  to  on. 

There  are  a  huge  number  of  reasonable  heuristics  for 
circuit  exchanges.  After  a  great  deal  of  experimentation 
with  many  of  these,  it  appears  that  the  choice  of  a 
particular  heuristic  is  not  critical.  Instead,  the  speed 
and  efficiency  with  which  potential  exchanges  can  be 
investigated  appears  to  be  the  linitinq  factor  affecting  t.-ie 
quality  of  the  final  design.  Finally,  as  the  size  of  the 
network  increases,  the  greater  the  cost  becomes  to  perform 
any  circuit  exchange  optimization.  Decomposition  of  the 
network  design  into  regions  becomes  necessarv  and  additional 
heuristics  are  needed  to  determine  effective  decompositions. 
It  presently  appears  that  these  methods  can  be  used  to 
de-.ign  relatively  efficient  networks  with  a  few  hundred  1,’lPs 
while  substantially  new  procedures  will  be  necessary  for 
networks  of  greater  size. 

The  topological  design  requires  a  routing  algorithm  to 
evaluate  the  throughput  capability  of  any  given  network. 
Its  properties  must  reflect  those  of  an  implencntablo 
routing  algorithm,  for  example,  within  the  AJd,/.;.TV. 
Althouqh  the  routing  problem  can  be  formulated  pk  a 
"multi commod i t y  flow  problem"  ClOj  and  solved  by  linear 
programming  for  networks  with  20  -  30  I'Ts  [B],  faster 
techniques  are  needed  when  the  routing  algorithm  is 
incorporated  in  a  design  procedure.  The  design  procedure' 
for  the  ARPA  Network  topology  iteratively  analyses  thousands 
of  networks.  To  satis fv  the  requirements  for  snnee,  an 
alqorithm  which  selects  the  least  utilized  patli  with  tne 
minimum  number  of  IMPS  was  initially  used  tl2J.  This 
alqorithm  was  later  replaced  by  one  which  sends  as  much 
traffic  as  possible  along  such,  paths  until  one  or  more 
circuits  approach  a  few  percent  of  full  utilization  [2Bj. 
These  highly  utilized  circuits  are  tiion  no  longer  allowed  to 
carry  additional  flow.  Instead,  now  paths  with  excess 
capacity  and  possibly  more  intermediate  nodes  are  found. 
The  procedure  continues  until  some  cutset  contains  only 
nearly  fully  utilized  circuits.  At  this  point  no  additional 
flow  can  be  sent.  For  design  purposes,  this  alqorithm  j  s  a 
highly  satis factorv  replacement  for  the  more  complicated 
multi-commodity  approach.  Using  the  algorithm,  it  has  been 
shov;n  that  the  throughout  capabilities  of  the  ARPA  Network 
arc  substantially  insensitive  to  the  distribution  of  traffic 
and  depend  mainly  only  on  the  total  traffic  flow  witnin  the 
network. 

3.3  ANALYTIC  MODELS  OF  NETWORK  PERrORJIANCK 

The  effort  to  determine  analytic  models  of  system 
performance  has  procedcu  in  two  phases:  (1)  the  prediction 
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of  average  time  delay  encountered  by  a  packet  as  it  passes 
through  the  network,  and  (2)  the  use  of  these  queueing 
models  to  calculate  optimum  channel  capacity  assignments  for 
minimum  possible  delay.  The  model  used  as  a  standard  for 
the  average  packet  delay  was  first  described  in  l21j .  where 
it  served  to  predict  delays  in  stochastic  communication 
networks.  In  C22J,  it  was  modified  to  describe  the  behavior 
of  ARPA-like  computer  networks  while  in  C23],  it  was  refined 
further  to  apply  directly  to  the  ARPANET. 

3.3.1  The  Single  Server  Model 

Queueing  theory  C20]  provides  an  effective  set  of 
analytical  tools  for  studying  packet  delay.  Much  of  this 
theory  considers  systems  in  which  messages  place  demands 
transmission  (service)  upon  a  single  communication  channel 
(the  single  server) .  These  systems  arc  characterized  by 
A(*f) ,  the  distribution  of  interarrival  times  betv/een  demands 
and  B (t ) ,  the  distribution  of  service  times.  When  the 
average  demand  for  service  is  less  than  the  capacitv  of  the 
channel,  the  system  is  said  to  be  stable. 

When  A  Of)  is  exponential  (i.e.  Poisson  arrivals),  and 
packets  are  transmitted  on  a  first-come-first-served  basis, 
the  average  time  T  in  the  stable  system  is 


T.  X? 

T  2(1 


+  t 


(1) 


where  X  is  the  average  arrival  rate  of  messages,  t  and  t 
are  the  first  and  second  moments  of  B(t)  respectively,  and 
If  the  service  time  is  also  exponential. 


T  = 


1  -  P 


(2) 


When  A(T)  and  B(t)  arc  arbitrary  distributions,  the 
situation  becomes  complex  and  only  weak  results  are 
available.  For  example,  no  expression  is  available  for  t, 
however  the  following  upper  bound  yields  an  excellent 

approximation  C193  as  f  ^  i* 
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where  *5.  and  are  the  variance  of  the  interarrival 

time  and  service  time  distributions,  respectively. 

3.3.2  Networks  Of  Queues 

Multiple  channels  in  a  network  environment  give  rise  to 
queueing  problems  that  are  far  more  difficult  to  solve  than 
single  server  systems.  For  example,  the  variability  in  the 
choice  of  source  and  destination  for  a  message  is  a  network 
phenomenon  which  contributes  to  delay.  A  principal 
analytical  difficulty  results  from  the  fact  that  flows 
throughout  the  network  are  correlated.  The  basic  approach 
to  solving  these  stochastic  network  problems  is  to  decompose 
them  into  analyzable  single-server  problems  which reflect 
the  original  network  structure  and  traffic  flow. 

Early  studies  of  queueing  networks  indicated  that  such 
a  decomposition  was  possible  £50,51};  however,  those  results 
do  not  carry  over  to  message  switched  computer  networks  due 
to  the  correlation  of  traffic  flows.  In  C21J  it  was  shown 
that,  for  a  wide  variety  of  communication  nets  the  length  of 
a  given  packet  could  be  considered  as  an  independent  random 
variable  as  it  passes  from  node  to  node.  Althouqh.  this 
"independence"  assumption  is  not  physically  realistic,  it 
results  in  a  mathematically  tractable  model  which  does  not 
seem  to  affect  the  accuracy  of  the  predicted  tine  delays. 
As  the  st  ''  and  connectivity  of  the  network  increases,  the 
assumption  becomes  increasingly  more  realistic.  With  this 
assumption,  a  successful  decomposition  which  permits  a 
channel-bv-channel  analysis  is  possible,  as  follows. 

The  packet  delay  is  defined  as  the  average  time  which  a 
packet  spends  in  the  network  from  its  entry  until  it  reaches 
its  destination.  The  average  packet  delay  is  denoted  as  T. 
Let  be  the  averacre  delay  for  those  packets  whose 

origin  is  IMP  j  and  whose  destination  is  IMP  k.  We  assume  a 
Poisson  arrival  process  for  such  packets  with  an  average  of 
V.  packets  per  second  and  an  exponential  distribution  of 
packet  lengths  with  an  average  oi  '/>*  bits  per  packet. 
With  these  definitions,  if  V  is  the  sum  of  tne 

quantities  *ok  ,  then  C21J 
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channel*"  dela™.  ^S^nrltTf  inc"^4’  /?,  ternS  °f  sl''gln 
for  the  ith  r-Lnnni  nrst  deflne  following  quantities 

af  as  tsr  L5«  as  “s  capacity  (bits/second) , 
/  /V  a.  the  average  packet  traffic  it  oarr-ir^ 

iaitinaS/?oC°ndM  T<  35  the  avtira^  tine  a  packet  speeds 
waiting  for  and  using  the  ith  channel .  B?  reiJtiS  the 

V'fJW  Via  the  Paths  selected  by'  the  Jout^ng 
algorithm,  it  is  easy  to  see  that  £21) 


£  —  T 

i  V  Ti 


(5) 


With  the  assumption  of  Poisson 
service  times,  the  quantities 
(2) .  For  an  average  packet  length 


traffic  and  exponential 
Tj_  are  given_by  equation 

4*  •  t=  ancd 


of 


(6) 


Thus  we  have  successfully  decomposed 
into  a  set  of  sinnle  single-channel 


the  analysis 
problems . 


problem 


A  refinement  of  the  decomoosition  permits  a 

;  pack°^  longt:’  attribution  and  uses  equation 
(1)  rather  than  eguation  (2)  to  calculate  T/  •  as  an 
approximadon,  the  Markovian  character  of  the  traffic  is 

wf^cluL  thfS«^;  Frh0rm0re'  ■ for  -nputornetiorbs 

i.r  ff.  .  .  effect  of  propagation  time  and  overhead 

deify  £22?23J  ***  follov'7incJ  equation  for  average  packet 
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Here,  At'  represents  the  average  length  of  a  Host  packet, 
and  'J/OL  represents  the  average  length  of  all  packets 
(including  acknowledgements ,  headers,  requests  for  next 
messages,  paritv  checks,  etc.)  within  the  network.  The 
expression  Vc*  KX./yC.)/(wC.  -  X.)UA  represents  thp  swranp 
packet  delay  onuie  ith  cnannel.  The  term  (X^/mC^)/(uC^  - 
is  the  average  time  a  packet  spends  waiting  at  tne  IMP  for 
the  ith  channel  to  become  available.  Since  the  packet  must 
compete  with  acknowledgments  and  other  overhead  traffic,  the 
overall  average  packet  length  appears  in  the 

expression.  The  term  '/uty  .the  time  required  to  transmit 
a  packet  of  average  length  .  Finally,  II  is  the  nodal 

processing  time,  assumed  constant  and  for  the  APvPA  IMP, 
approximately  equal  to  0.35  ms?  is  the  propagation 

tine  on  the  ith  channel  (about  20  ms  for  a  3000  mile 
channel) . 

-  Assuming  a  relatively  homogeneous  set  of  Cj  and 

?*  ,  no  individual  term  in  the  expresson  for  delay  will 
dominate  the  summation  until  the  flow  in  one  channel  (say 
channel  j£0  )  approaches  capacity.  At  that  point,  the  term 

7*  ,  and  hence  T  will  grow  rapidly.  The  expression  for 

delay  is  then  dominated  by  one  or  more  terns  and  exhibits  a 
threshold  behavior.  Prior  to  this  threshold,  T  remains 
relatively  constant. 

The  accuracy  of  the  time  delay  model,  as  well  as  this 
threshold  phenomenon  was  demonstrated  on  a  10  node  network 
fl4j  and  on  the  ten  node  AP.PA  net  derived  from  Figure  1(c) 
by  deleting  the  rightmost  five  IMPs.  Using  the  routing 
procedure  described  in  the  last  section,  C28]  and  eguai 
traffic  between  all  node  pairs,  the  channel  flows  Ac  were 
found  for  the  ten  node  net  and  the  delay  curves  shown  in 
Figure  3  were  obtained.  Curve  A  was  obtained  with  fixed 
1000  bit  packets*,  while  curve  B  was  generated  for 
exponentially  distributed  variable  length  packets  with 
average  size  of  500  bits.  In  both  cases  A  and  B,  all 
overhead  factors  were  ignored.  Note  that  the  delay  remains 
small  until  a  total  throughput  slightly  greater  than  400 


*In  case  A,  the  application  of  equation  (1)  allows 
constant  packet  lengths  (i.e.  zero  variance). 


for 
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kilobits/second  is  reached.  The  delay  then  increases 
rapidly.  Curves  C  and  D  respectively  represent  the  same 
situations  when  the  overhead  of  136  bits  per  packet  and  per 
RFNM  and  152  bits  per  acknowledgment  are  included.  Notice 
that  the  total  throughput  per  IMP  is  reduced  to  250 
kilobits/second  in  case  C  and  to  approximately  200 
kilobits/second  in  case  D. 

In  the  same  figure,  we  have  illustrated  with  x's  the 
results  of  a  simulation  performed  with  a  realistic  routing 
and  metering  strategy.  The  simulation  omitted  all  network 
overhead  and  asuned  fixed  lengths  of  1000  bits  for  all 
packets. 

It  is  difficult  to  develop  a  practical  routing  and  flow 
control  procedure  that  will  allow  each  IMP  to  input 
identical  amounts  of  traffic.  To  compare  the  delay  curve  A 
with  the  points  obtained  by  simulation,  the  curve  should 
actually  be  recomputed  for  the  slightly  skewed  distribution 
that  resulted.  It  is  notable  that  the  delay  estimates  from 
the  simulation  (which  used  a  dynamic  routing  strategy)  and 
the  computation  (which  used  a  static  routing  strategy  and 
the  time  delay  formula)  are  in  close  agreement.  In 
particular,  they  both  accurately  determined  the  vertical 
rise  of  the  delay  curve  in  the  range  just  above  400 
kilobits/second,  the  formula  by  predictina  infinite  delay 
and  the  simulation  by  rejecting  the  further  input  of 
traffic. 

In  practice  and  from  the  analytic  and  simulation 
studies  of  the  ARPANET  ,  the  average  queueing  delay  is 
observed  to  remain  small  (almost  that  of  an  unloaded  net) 
and  well  within  the  design  constraint  of  0.2  seconds  until 
the  traffic  within  the  network  approaches  the  capacity  of  a 
cutset.  The  delay  then  increases  rapidly.  Thus ,  as  long  as 
traffic  is  low  enough  and  the  routing  adaptive  enough  to 
avoid  the  premature  saturation  of  cutsets  by  guiding  traffic 
alonff  paths  with  excess  capacity,  queueing  delays  are  not 
significant. 

3.3.3  Continuous  Capacity  Optimization 

One  of  the  most  difficult  design  problems  is  the 
optimal  selection  of  capacities  from  a  finite  set  of 
options.  Although  there  are  many  heuristic  approaches  to 
this  problem,  analytic  results  are  relatively  scarce.  (For 
the  specialized  case  of  centralized  networks,  an  algorithm 
yielding  optimal  results  is  available  fllj).  While  it  is 
possible  to  find  an  economical  assignment  of  discrete 
capacities  for,  say,  a  200  IMP  network,  verv  little  is  known 
about  the  relation  between  such  capacity  assignments, 
message  delay,  and  cost. 
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To  obtain  theoretical  properties  of  optimal  capacity 
assignments,  we  first  ignore  the  constraint  that  capacities 
are  obtainable  only  in  discrete  sizes.  In  C21J  the  problem 
was  posed  where  the  network  topology  and  average  traffic 
flow  were  assumed  to  be  known  and  fixed  and  an  optimal  match 
of  capacities  to  traffic  flow  was  found.  Also,  the  traffic 
was  assumed  to  be  Markovian  (Poisson  arrivals  and 
exponential  packet  lengths)  and  the  independence  assumption 
and  decomposition  method  were  applied.  For  each  channel, 

the  capacity  Cji  was  found  which  minimized  the  average 

m®ssage  delay  T,  at  a  fixed  total  system  cost  D.  (Since 

is  the  average  bit  rate  on  the  ith  channel,  the 
solution  to  any  optimal  assignment  problem  must  provide  more 
than  this  minimal  capacity  to  each  channel.  This  is  clear 
since  both  equations  (6)  and  (7)  indicate  that  7^  will 
become  arbitrarily  large  with  less  than  (or  equal  to)  this 

amount  of  capacity.  It  is  not  critical  exactly  how  the 

excess  capacity  is  assigned,  as  long  as  "T  .)  The 

optimization  further  assumed  that  a  total  of  D  dollars  was 
available  to  provide  the  channel  capacities  and  that  the 
cost  of  the  ith  channel  was  linear  at  a  rate  of  oi/ 
dollars  per  unit  of  channel  capacity;  that  is 

•  The  simpler  form  for  7^  in  equation 
(6)  is  used  in  this  formulation  and  T  is  as  given  in 
equation  (5). 


The  solution  to  this  problem  assiqns  a  capacity  to  the  ith 
channel  in  an  amount  equal  to  plus  some  excess 
capacity  proportional  to  the  square  root  of  that  traffic. 
With  T  evaluated  for  this  assignment. 


T  = 


(8) 


Here 

packets 

between 

channel 


A**  £  represents  the 

flow  within  the  net  and 
D  and  the  amount  which  must  be 
with  capacity  ,  namely 


'f 


total  rate  at  which 
Pc  is  the  difference 
spent  to  provide  each 
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Moreover,  *  y  is  easily  shown  to  represent  the  average 
path  length  for  a  packet. 

If  4 !c  -  1  for  all  channels,  t>-  £<l’  c  where  C 
represents  the  total  capacity  within  the  network.*  In  this 
case , 


T  "  tied"-  np)  (^/7r)2  a0> 

Here  f  =  is  the  ratio  of  the  rate  at  which  bits 

enter  the  network  to  the  rate  at  which  the  net  can 

handle  bits,  the  quantity  j~  represents  a  dimensionless 
form  of  network  "load."  As  the  load  £  approaches  , 

the  delay  T  grows  very  quickly,  and  this  point  f  =  '/* 
represents  the  maximum  load  which  the  network  can  support. 
If  capacities  are  assigned  optimally,  all  channels  saturate 
simultaneously  at  this  point.  In  this  formulation  n  is  a 
design  parameter  which  depends  upon  the  topology  and  the 
routing  procedure,  while  P  is  a  parameter  which  depends 
upon  the  input  rate  and  the  total  capacity  of  the  network. 
Equation  (10)  provides  insight  into  topological  structure 
and  routing  procedures  C21J. 

In  a  recent  paper  £26]  ,  it  was  observed  that,  in 
minimizing  T,  a  wide  variation  was  possible  among  the  packet 
delays  T '<C  .  As  a  result,  the  problem  of  finding  the  sets 
of  channel  capacities  which  minimize  was  considered, 

where 


The  solution  for  the  optimal  channel  capacity  assignment 
with  a  given  value  k,  denote  by 

♦The  assumption  <5^  =  1  is  of  practical  importance  in 
the  case  of  satellite  channels  £33]. 
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With  this  capacity  assignment, 


(13) 


Note  that  the  assignment  C ^  is  the  previously  mentioned 
assignment  which  minimizes  T,  and  is  the  previously 
stated  value  of  delay.  As  k  increases  the 
variation  in  the  y  decreases  and  as  k  — - 


(14) 


In  the  limit,  the  channel  capacity  is  assigned  to  give  each 
channel  its  minimum  required  amount  plus  a  constant 

additional  amount.  All  the  Tc  are  equal  and 
_  .  Moreover,  setting  k  =  0  yields 


—  Id. 
uD  .  j 


nydi 


(15) 


For  this  assignment,  the  value  of  T  is  identical  to  the 
value  it  achieves  when  k  — ,  although  different  channel 
capacity  assignments  typically  occur  at  these  extremes  C46], 
If  all  =1  a  channel  capacity  is  assigned  in  proportion 
to  the  traffic  carried  by  that  channel  (commonly  known  as 
the  proportional  capacity  assignment) .  Although  the  value 
of  T  is  minimized  for  the  capacity  assignments  which 
result  when  k  =  1,  T  increases  slowly  as  k  varies  from  unity 
and,  moreover,  the  variance  of  packet  delay  is  minimized 
when  k  =  2. 

In  studying'  the  ARPANET  C23J  a  closer  representation  of 
the  actual  tariffs  for  high  speed  telephone  data  channels 
used  in  that  network  was  provided  by 
setting  D  =  /  d.C,01  where  0  /  •(  5  1.*  This 
approach  requires  mthcr  solution  of  a  non-linear  equation  by 
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numerical  tecnniques.  On  solving  the  equation,  it  can  be 
shown  that  the  packet  delay  T  varies  insignificantly 
with  o<v  for  .3  £  d?  1.  This  indicates  that  the 
closed  form  solution  discussed  earlier  with  o(  =  1  is  a 
reasonable  approximation  to  the  more  difficult  non-linear 
problem. 

In  practice,  the  selection  of  channel  capacities  must 
be  made  from  a  small  finite  set.  Although  some  theoretical 
work  has  been  done  in  this  case  by  approximating  the 
discrete  cost— capacity  functions  by  continuous  ones,  much 
remains  to  be  done  £13,25].  Because  of  the  discrete 
capacities  and  the  time  varying  nature  of  network  traffic, 
it  is  not  generally  possible  to  match  channel  capacities  to 
the  anticipated  flows  within  the  channels.  If  this  were 
possible,  all  channels  would  saturate  at  the  sane  externally 
applied  load.  Instead,  capacities  are  assigned  on  the  basis 
of  reasonable  estimates  of  average  or  peak  traffic  flows. 
It  is  the  responsibility  of  the  routing  procedure  to  allow 
the  traffic  to  adapt  to  the  available  capacity  E14).  Often 
two  IMP  sites  will  engage  in  heavy  communication  and  thus 
saturate  one  or  more  critical  network  cutjets.  In  such 
cases,  the  routing  will  not  be  able  to  send  additional  flow 
across  these  cuts.  The  network  will  therefore  experience 
"premature"  saturation  in  one  or  a  snail  set  of  channels 
leading  to  the  threshold  behavior  described  earlier. 


*0f  course  the  tariffs  reflect  the  discrete  nature  of 
available  channels.  The  use  of  the  exponent  provides 
a  continuous  fit  to  the  discrete  cost  function.  For  the 
ARPANET,  *  .3. 
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IV.  DISCUSSION 


A  major  conclusion  from  our  experience  in _ 
desiqn  is  that  messaqe  switched  networks  of  the  ARPA  tyt  c 
are  no  lonqer  difficult  to  specify.  They  may  be  lmnlemcnte 
straightforwardly  from  the  specifications,  they  can  ^  ^ 
exnensive  than  other  currently  available  technical 

approaches;  they  perform  remarkably  well  as  a 

communication  system  for  interconnecting  t^-sharino  and 
batch  processing  computers  and  can  be  adapted  to  directl 
haSSo  tempos?  aisolaya  and  many  other  kinds  of  terminal 
devices  and  data  processing  equipment 

The  principal  tools  available  for  the  design  of 
netv;or!:s  are  analysis,  simulation,  heuristic  procedures ,  an 
experimentation.  Analysis,  simulation  and  heuristics  iavo 
been  the  mainstays  of  the  work  on  modeling  and  topological 
optimization  while  simulation,  heuristic  procedures  and 
experimental  techniques  have  been  the  major  for  the 

actual  network  implementation.  Experience  has  shown  that 
all  of  these  methods  are  useful  wnile  none  are  all  powerful. 
The  most  valuable  approach  has  been  the  simultaneous 
several  of  these  tools. 

Eac  approach  has  room  for  considerable  improvement. 
The  analysis  efforts  have  not  vet  yielded  results  in  many 
important  areas  such  as  routing.  However,  for  prediction  of 
Say  ttis  approach  loads  to  a  simple  threshold  model  which 

itl1”S£erC^omet“<1pr  lef  “at ’S  *  .  * 

^r^inCp^r  r—Kf  cr  .K?r 

deal  of  additional  work  is  required  for  networks  of  jreat  r 

size  Simulation  is  a  well  developed  tool  that  is 

expensive  to  apply  and  limited  in  the  overall  understanumg 

t-h^t  it  tan  vicld.  For  these  reasons,  simulation  appears  to 

be  most  useful  only  in  validating  models,  and  in  assisting 

in  detailed  design  decisions  such  as  the  number  of  buffers 

t-hst-  an  imp  should  contain.  As  tnc  size 

continue  to  grow,  it  appears  that  simulatronwril  become 
virtually  useless  as  a  total  design  tool.  The  ultimate 
standard”  by  which  all  models  and  conclusions  can  be  teste  . 
is  experimentation.  Experimentation  with  the  actual  network 
iSs  "conceptually  relatively  straightforward  and  very  useful. 
Although  experiments  arc  often  loaisticnlly  ^ 

perform/  they  can  provide  an  easy  means  for  testing  models, 

heuristics  and  desiqn  parameters. 

The  outstanding  design  problems  currently  facing  tne 
network  designer  are  to  specify  and  determine  the  pronert 
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of  the  routing,  flow  control  and  topological  structure  for 
large  networks.  This  specification  must  make  full  use  of  a 
wide  variety  of  circuit  options.  Preliminary  studies 
indicate  that  initially,  the  most  fruitful  approaches  will 
be  based  on  the  partitioning  of  the  network  into  regions ,  or 
equivalently,  constructing  a  large  network  by  connecting  a 
number  of  regional  networks.  To  send  a  message,  a  Host 
would  specify  both  the  destination  region  and  the 
destination  IMP  in  that  region.  No  detailed  implementation 
of  a  large  network  has  yet  been  specified  but  early  studies 
of  their  properties  indicate  that  factors  such  as  cost, 
throughput,  delay  and  reliability  are  similar  to  those  of 
the  present  ARPANET,  if  the  ARPA  technology  is  used  [9]. 

Techniques  applicable  to  the  design  of  large  networks 
are  presently  under  intensive  study.  These  techniques 
appear  to  split  into  the  same  four  cateqories  as  small 
network  design  but  approaches  may  differ  significantly.  For 
example,  large  nets  are  likely  to  demand  the  placement  of 
high  bandwidth  circuits  at  certain  key  locations  in  the 
topology  to  concentrate  flow.  These  circuits  will  require 
the  development  of  a  high  soeea  IMP  to  connect  them  into  the 
net.  It  is  likely  that  this  high  speed  IMP  will  have  the 
structure  of  a  high  speed  multiplexor,  and  may  require 
several  cooperating  processors  to  obtain  the  needed  computer 
power  for  the  job.  Flow  control  strategies  for  larce 
networks  seem  to  extrapolate  nicely  from  small  network 
strategies  if  each  region  in  the  large  network  is  viewed  as 
a  node  in  a  smaller  network.  However,  this  area  will 
require  additional  study  as  will  the  problem  of  specifying 
effective  adaptive  routing  mechanisms.  Recent  efforts 
indicate  that  efficient  practical  schemes  for  small  networks 
will  soon  be  available.  These  schemes  seen  to  be  applicable 
for  adaptive  routinq  and  flow  control  in  networks 
constructed  from  regional  subnetworks.  The  development  of 
practical  algorithms  to  handle  routing  and  flow  control  is 
still  an  art  rather  than  a  science.  Simulation  is  useful 
for  studying  the  properties  of  a  given  heuristic,  but 
intuition  still  plays  a  dominant  role  in  the  system  desion. 

Several  open  questions  in  network  desion  presently  are: 
(1)  what  structure  should  a  high  bandwidth  IMP  have;  (2)  How 
can  full  use  be  made  of  a  variety  of  high  bandwidth 
circuits;  (3)  How  should  large  networks  be  partitioned  for 
both  effective  design  and  operation;  and  (4)  what 
operational  procedures  should  largo  networks  follow.  Much 
work  has  already  been  done  in  these  areas  but  much  more 
remains  to  be  done.  We  expect  substantial  progress  to  be 
achieved  in  the  next  few  years,  and  accordingly,  tne 
increased  understanding  of  the  properties  of  messaqe 
switched  networks  of  all  sizes. 
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THE  FLOW  DEVIATION  METHOD:  AN  APPROACH  TO 
STQRE-AND-FOEWARD  OOMCNICATIOJ  NETWORK  CESIGJ* 

L.  Fratta,  M.  Gerla  and  L.  Kleinrock 

ABSTRACT 

A  "flow  deviation"  method  for  the  s\±>-cptimization  of  nultioonnodity 
flow  in  networks  is  described.  The  method  is  very  general  but  proves  to 
be  particularly  fast  and  successful  in  stoie-and- forward  (S/F)  network 
problems.  The  flow  deviation  (ID)  method  for  networks  is  quite  similar 
to  the  gradient  method  for  continuous  functions;  here  the  concept  of  gradient 
is  replaced  by  the  concept  of  "marginal"  shortest  path.  As  in  the  gradient 
method/  the  application  of  successive  flow  deviations  leads  to  a  local  opti¬ 
mum.  When  the  solution  contains  several  local  optima,  randomization  of  the 
starting  flow  configuration  atteirpts  to  locate  them.  TWo  interesting  appli¬ 
cations  relative  to  the  ARPA  Conputer  Network  are  presented. 


*This  work  was  supported  by  the  Advanced  Research  Projects  Agency  of 
the  Department  of  Defense  (DAHC15-69-C-0285) . 
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THE  FLOW  DEVIATION  METHOD:  AN  APPROACH  TO 
STDRE-AND-FOFWARD  COMMUNICATION  NETWORK  DESIGN* 

L.  Fratta,  M.  Gerla  and  L.  Kleinnock 


1.  Introduction 

In  this  paper  we  consider  a  sub-optimal  procedure  (the  " f low-deviation" 
method)  for  assigning  flew  within  store— and— forward  communication  networks 
so  as  to  reduce  cost  and/or  delay  for  a  given  topology  and  for  given  external 
flow  requirements.  We  begin  by  defining  the  basic  model  belcw  and  follow 
that  with  seme  examples.  We  then  discuss  various  approaches  to  the  problem 
and  then  introduce  and  describe  the  "flow  deviation"  method.  This  method 
is  evaluated  under  same  further  restrictions  and  is  then  applied  to  various 
problem  formulations  for  the  AREA  network  [6]  ,  f/] . 

We  consider  a  collection  of  nodes  ,  (i  =  1,  ...  m)  called  sources, 
and  a  collection  of  nodes  Dj,  (j  —  1,  •••  W  t  called  destinations.  We 
are  required  to  route  a  quantity  r^j  of  type  (i,  j)  commodity  from  source 
Si  to  destination  Dj,  throu^i  a  given  network  (Fig.  1) . 

The  multicommodity  flow  problan  consists  of  finding  the  optimal  routing 
of  all  such  oanmodities,  which  minimizes  (or  maximizes)  a  well-defined  per¬ 
formance  function  (e.g.,  cost  or  delay),  such  that  a  set  of  constraints  (e.g. , 
channel  capacity  constraints)  are  satisfied. 


♦This  work  was  supported  by  the  Advanced  Research  Projects  Agency  of 
the  Department  of  Defense  (DAHC15-69-C-0285) . 
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Mora  generally,  consider  a  network  of  n  nodes  N^,  (i  ■  1,  ...  n),  anc 

suppose  that  a  required  quantity  r.  .  of  type  (i,j)  ccrmodity  umt  be 

**"  J 

sent  from  to  Nj  for  all  i  and  j;  this  more  general  case  can  be 
reduced  to  the  first  cne  by  setting  certain  of  the  r. .  to  zero  (Fig.  2) 


Figure  2 


t  I i 


The  jj^^ccmmodity  problem  can  formally  be  expressed  in  the  following  way: 


Given:  -  a  network  of  n  nodes  and  b  branches 

-  an  n  x  n  matrix  R  =  [r^  ] ,  called  the  requirenent 

matrix /Whose  entries  are  non-negative. 

Minimize  (or  naximize)  ^  P(F) , 
over  F  . 

where  F  is  the  flew  configuration  and  P  is  a  well- 
defined  performance  function. 

Furthermore,  F  must  satisfy  the  following  constraints: 


Constraints: 


1.  F  mist  be  feasible.  Two  conditions  are  required  for 
feasibility: 

Condition  a:  if  the  arcs  of  the  network  are  "directed," 
must  have 


we 


fijk£  -  0 


Vi,j,k ,% 


where  f^^  is  the  portion  of  conmodity  (i,j) 
flowing  on  the  directed  arc  (k, £) .  in  other  words, 

(i/j)  can  be  routed  only  on  directed  paths 

from  N.  to  N.. 

1  3 


^Without  loss  of  generality,  only  the  min.  problem  is  considered  in 
the  following. 

2  .  _ 

The  multicommodity  flow  configuration  F  is  given  in  terms  of  f 

ijk£ 

vfe  note  that  F  may  be  also  described  by  a  vector  F  =  (f  ,f  ...  f  ) 

12  b 

where  b  is  the  total  number  of  arcs  and 

n  n 

*m-  E  E*<.M 

i=l  j=l  ~jJ06 

where  we  have  labelled  the  arcs  in  such  a  way  that  arc  (k,£)  is  labelled 
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Condition  b: 

r.  . 

if  i  =  j 

ID 

n  | 

-r.  . 
ID 

if  i  =  i 

0 

otherwise 

In  other  words,  this  condition  inplies  the  conserva¬ 
tion  of  the  flows  (cxxtmodity  by  oonrodity)  at  each 
node  and  guarantees  the  required  flew  between 

N.  and  N.. 
i  D 

2.  F  most  satisfy  some  additional  constraints,  different  from 

problem  to  problem  (e.g.,  capacity  constraints  on  each  channel 
or 

and /tost  constraints). 


m  and  f  is  the  total  flow  (of  all  cormodities)  on  branch  m.  As  a  third 

representation  for  F  we  may  identify  the  routes  taken  by  all  portions  of 

the  flow  requir orient  r^  in  travelling  from  Ni  to  Nj  for  all  i  and  j 

That  is,  suppose  fron  Ni  to  N.  there  are  s  such  routes.  Then  route 

it  has  weight  a,  if  it  carries  a  fraction  a,  of  the  requirement 
ii.k  k  ^ 


!„  the  follow^  we  use  whichever  of  these  representations  is  tost  convenient 
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2.  Examples  of  multi cxmmodity  problems 

Let  us  consider  as  an  example  a  store-and— forward  (S/F)  ccnirunicaticn 
network  [1] .  In  such  a  network,  messages  travelling  from  Ni  to  are 
"stored"  in  queue  at  any  intermediate  node  N^,  white  awaiting  transmission, 
and  are  sent  "forward"  to  N^,  the  next  node  in  the  route  from  N±  to  N., 
when  channel  (k,A)  permits.  Thus,  at  each  node  there  are  different  queues, 
cne  for  each  output  channel.  The  message  flew  requirements  between  nodes 
arise  at  randan  times  and  the  messages  are  of  randan  lengths,  therefore  the 
flaws  in  the  channels  and  the  queue  lengths  in  the  nodes  are  randan  variables. 
Under  appropriate  assumptions3  an  analysis  of  the  system  can  be  carried  out 
[1];  in  particular,  it  is  possible  to  relate  the  average  delay  T  suffered 
by  a  message  travelling  frcm  source  to  destination  (the  averac^  is  over  time 
and  over  all  pairs  of  nodes)  to  the  average  flows  in  the  channels. 

The  result  of  the  analysis  is: 


(1) 


b 

=  Z 

i=l 


l 

y 


where 


Y  * 


£ 


£ 


id  j=l 


T  =  total  average  delay  per  message  [sec/Snessg] 
b  =  #  of  arcs  in  the  network 

—  message  rate  on  channel  i  [messg/sec] 
rij  =  tota^  n®333^  arrival  rate  from  external  sources  [messg/sec] 
T.  =  average  delay  suffered  by  a  message  on  channel  i  [sec/nessg] 


3 Assumptions:  Poisson  arrivals  at  nodes,  exponential  distribution  of 
ires  sage  length,  independence  of  arrival  processes  at  different  nodes,  inde¬ 
pendence  assumption  of  service  times  at  successive  nodes  [1]  • 
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T.  as 

i 


the  sum  of  two  ccrrponents: 


where 


and 


rpi  — - 

1  MC,  -  A. 


T  =  T!  +  TJ 
1  1  X 

1 _  _  transmission  and  qiiueing  delay 


rj.u  -  =  propagation  delay 

c.  =  capacity  of  channel  i  [bits/sec] 


l/j^  -  average 


nessage  length  [bits/messg] 


We  can 


(I1) 


rewrite  (1)  as  follows: 


, .  \  /  -  f  .  (1*)  becomes: 

Letting  A  i/f*"  ri 


(1") 


where  : 


T  =  i  Z 
y  r 


r%-r  +  fiP- 

ci  ti  i 


f .  =  bit  rate  on  channel  i  tbits/sec] 


i 

>•  = 


Pi  =/*Pi 


The  average 


aelay  T  is  the  rest  corenon  performance  reasure  for  S/P 
nulticornodity  prcblm  consists  of  finding  that  routang. 


networks ,  and  the 
or  flew  pattern  F.  «hidt  minimizes  T. 
We  may  new  pose  tiro  problems  . 
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Problem,  A:  "Routing  assignment" 

given:  topology,  channel  capacities  and  a 

requirement  mtx  R 


minimize,  T(Fi  „  I 
"over  F'  Y 


y  fr&r +  *’  £i 


constraints: 

(i)  F  feasible 
(ii)  1  s  1#  •**  b 

•n*  problem  Is  in  the  standard  mlticomrfity  form  and  the  addiUHSl  con- 

straints  are  opacity  constraints. 

A  second  interesting  problem  in  S/F  networks  is  forsnUted  below. 
A--,  that  we  have  a  given  network  topology  in  whith  the  channel 
capacities  have  to  be  assigned.  A  cost  is  associated  with  the  values  of 
the  capacities r  and  the  total  cost  of  the  network  is  given.  In  additim, 
the  flow  routes  mist  be  determined. 

The  problem  statement  is: 

Problem  B:  "Routing  and  capacities  assignment,  general  cost-cap 

given;  topology,  requirement  mtx  R,  nmber  of 

dollars  available  D 


minimize, 
over  C,F* 


S'.C,F)  =  i  £  (-57  -  IT  +  pi’  fi 

I  1=1  1  1 


constraints : 


(i)  F  feasible 

(ii)  ^  £C.,  i  =  If  - 

(iii)  E  d.(C.)  *  D 
i=l  1 
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where 
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C=  {CyC2  •••  cy 

d^ (C)  =  cost-cap.  function  for  arc  i 

The  minimization  can  be  carried  out  first  on  C,  keeping  F  fixed,  and 
then  on  F. 

If  the  cost-cap .  functions  are  linear  (i.e.,  d^C)  =  d^C.),  then  the 
minimization  over  C  can  easily  be  performed  by  the  method  of  Lagrange 
multipliers  and  we  get  the  optimum  capacities  as  functions  of  the  flows  [1] 


(3) 


where 


Since 


and 


then 


and 


b 

•D-  E 

i=l 


fidi 


b 

D  >  E  d.C.  for  (iii) 
“  i=l  1  1 


b  b 

E  d.C.  >  £  for  (ii) 

i=l  11  i=l  1  1 


b 

D>  E 

i=l 


difi 


b 

D  =  D  -  D  d .  f .  >  0  (iv) 

e  i=l  1  1 
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It  is  easy  to  see  from  (3)  that  (iv)  implies  also  (ii)  and  (iii)  ,  hence 
both  (ii)  and  (iii)  can  be  replaced  by  (iv) . 

By  introducing  the  expression  of  the  optimum  capacities  given  by  (3) 

into  (2) ,  we  obtain; 

Problon  B;  "Routing  and  capacities  assignment,  linear  cost-cap.  function" 


qiven: 


minimize. 
over  F  * 


topology,  requirement  mtx  R,  #  of  dollars  D 


£fi?i 


constraints; 

(i)  F  feasible 
(ii)  De  >  0 


Again  the  problem  is  reduced  to  an  optimal  flew  problem  of  the  standard 
multioanrcdity  form.  The  additional  constraint  is  new  a  cost  constraint. 
These  two  exanples  will  be  referred  to  in  the  sequel  as  Problem  A  and  Prob¬ 
lem  B,  and  will  be  further  investigated  at  the  end  of  the  paper. 
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3.  Approaches  to  the  Multicommodity  Problem  for 
Some  Special  Cases 

Linear  programming  approach 

a.  With  a  linear  objective  function  and  linear  constraints,  the  problem 
can  be  solved  with  linear  programming  techniques  by  taking  advantage  of 
the  decomposition  principle  [9] . 

Any  feasible  flow  F  can  be  expressed  as  a  convex  combination  of 

4 

w  non-bifurcated  feasible  flows  §^,  (i  =  1,  ...  w),  i.e.: 

w 


■  I 


i=l 


a.®  . 
1*1 


w 

where  £  =  1  •  It  can  be  shown  that  w  <  b  +  1,  where  b  is  the 

i=l  1 

number  of  arcs  [9] .  Suppose  that  the  's  are  known,  then  the  variables 
of  the  linear  program  are  the  {a},  and  the  feasibility  constraint  on  F 
is  replaced  by: 


The  number  of  possible  candidates  ^  for  the  optimal  solution  is  enormous  5; 
however,  during  the  application  of  the  revised  simplex  method,  only  a  snail 


A  flew  configuration  is  non-bifurcated  if  commodity  (i,j)  is  flowing 
on  one  path  only,  V  i,  j. 

^tt  is  equal  to  the  number  of  all  the  combinations  of  paths  from  i  to 

j*  V  i,j- 
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number  of  candidates  is  considered,  using  shortest  path  criteria  in  the 
selection  of  the  pivot  column.5'  The  optimal  solution  can  be  expressed 
as  a  set  of  and  a^,  i  =  1,  .  •  • »  w. 

b.  With  a  non-linear  performance  function,  but  still  with  linear  con¬ 
straints,  the  decomposition  method  can  be  applied  recursively  by  linearizing 
the  performance  function  around  the  new  optimal  solution  at  each  iteration. 
The  procedure,  towever,  often  becomes  very  time-consuming. 

We  make  two  useful  observations  at  this  point: 

Observation  #1: 

The  multicommodity  problem,  as  it  appears  in  standard  form, 
is  a  constrained  optimization  problem  and  constrained  opimization 
methods  (e.g.,  linear  programing)  are  therefore  required.  'However, 
if  we  represent  the  flow  F  as  a  convex  cxmbination  of  flows 
with  weights  a.,  ^  feasibility  constraint  on  F  is  replaced 

by  the  nnnvexitv  constraint  ^cu  =  1.  Furthermore,  if  we  introduce 
appropriate  penalty  functions  which  account  for  the  additional  con¬ 
straints  (e.g.,  capacity  or  cost  constraints  as  shown  belcw) ,  we 
arrive  at  a  problem  where  the  only  remaining  constraint  is: 


-’'private  carmunicaticn 

matics,  UCLA. 


from  Professor  D.  Cantor,  Department  of  Mathe- 
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It  will  be  shown  later,  however,  that  in  many  practical  cases  the 
optimum  flow  can  be  approximated  with  a  single  nan-bifurcated  flew; 
thus,  all  c^'s  are  zero  but  one,  which  is  unity,  and  the  constraint 
can  be  disregarded.  Ihe  problem  becomes  an  unconstrained  problem, 
and  methods  of  unconstrained  optimization,  similar  to  the  gradient 
method,  much  faster  than  the  constrained  ones,  are  available.6 7 

Oh  the  other  hand,  if  the  optimal  solution  cannot  be  approxi¬ 
mated  by  only  one  non-bifurcated  flew,  and  mare  than  cne  route  must 

be  considered  between  each  pair  of  nodes,  than  the  problem  can  be 

.  .  7 

successfully  solved  with  Lagrange  multipliers. 

Observation  #2; 

Consider  Problem  A  and  Problen  B,  two  typical  S/F  problems. 

One  notices  that  the  performance  function  T  goes  to  00  When 
the  flow  F  approaches  the  boundary  defined  by  the  additional  con¬ 
straints  (i.e. ,  when  any  channel  becomes  saturated  in  Problem  A, 
or  when  the  excess  De  reduces  to  zero  in  Problem  B) . 

Ihis  result  is  quite  general  for  S/F  networks,  because,  when 
F  approaches  the  bourdaries  of  the  constraint  set,  sane  saturation 
usually  occurs,  the  queues  at  the  nodes  grew  large  and  the  delay  T 
increases  rapidly.  Thus,  in  the  S/F  case,  the  penalty  functions 
mentioned  in  Observation  #1  are,  as  a  rule,  already  incorporated  in 
the  performance  measure. 

6Of  course,  we  are  still  faced  with  the  problen  of  finding  a  permissible 
starting  flow  configuration;  the  problen  is  analogous  to  the  Phase  I  of  the 
linear  programing  [9]  and  will  be  discussed  later. 

7We  are  now  developing  at  UOA  an  algorithm  that  will  take  care  of  this 
more  general  case. 
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If  the  above-mentioned  properties  hold,  then  the  multiccrrmodity 
problem  becomes  an  unconstrained  optimization  problem  and,  as  such, 
can  be  solved  by  the  flew  deviation  (FD)  method,  an  unconstrained 
optimization  method. 

4.  The  Philosophy  of  FD  Method 

Suppose  we  have  successfully  transformed  the  original  multi catmodity 
problem  into  an  unconstrained  minimization  problem.  Vfe  may  then  apply  the 
FD  method,  an  uncons tr ained  optimization  method  which,  in  many  respects, 
resembles  the  gradient  method.  In  order  to  point  out  the  similarities, 
let  us  apply  the  gradient  method  to  the  following  minimization  problan: 


(continued) 
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rain  f  (x)  ,  x  £  X 


where  X  is  an  n-diroensicnal  vector  space.  If  f(x)  is  continuous  and 
differentiable  in  X,  and  the  n  first  partial  derivatives  are  continuous, 
then  the  gradient: 


Vf(£> 


where 


7  -  (—  —  JL  ) 

V  =  8x.  '  3x0  '  *•*'  3x  ' 
12  n 


gives  the  direction  of  maxiraura  increase  of  f(x)  at  x^. 

As  we  have  a  minimization  problem,  the  gradient  method  consists  of 
of  proper  size 

taking  a  step /In  the  direction: 


-Vf(x) 


x  = 


So 


In  the  network  case,  the  flow  F  corresponds  to  the  variable  x. 

Suppose  for  simplicity  that  F  is  a  non-bifurcated  flew.  Then  F  may 
be  described  as  a  collection  of  routes,  one  for  each  pair  of  nodes.  let 
ir.j  represent  the  route  between  and  Nj.  Then  the  performance  function 
can  be  expressed  as: 


P(F)  =  P  ({^ij  >) 

Hie  "marginal  Shortest  Path  (SP)  matrix,"  which  will  be  defined  later, 

,  of  new  routes,  m'ij, 

provides  a  collection  of  {ir^ },  /  the  paths  of  nfixiirum  decrease  of  P(F) 
for  infinites  imal  deviation  of  flows.  In  other  words,  P(F)  decreases 


13 


8!) 


„t  rapidly  by  deviating  an  infinitesimal  portion  of  the  cmncdity  U.J) 

from  Tf  to  H'L  foraU  i  and  j. 

T  J  the  nargit.1  SP  mu  in  networ*  Co.  has  the  sane  interpreted 

as  the  gradient  for  a  function  of  the  continuous  variable  x. 

m  the  nost  general  case  of  a  prrt,l«  with  several  local  mirdma,  the 
perfontanee  functien  P«  can  be  ^statically  represented  as  in  Frg.  3. 


0  permissible  starting  solutions 
X  local  optima 

Figure  3 

f  frwiina  ^nr/jS^bl^tarting  configuration  for 
Our  mpthr'i.  consists  of  finding  / 

the  flow  F  and  then  applying  suooessive  H>'s  until  a  local  optiiem  is 
reached. 

m  the  oase  of  several  local  optina,  a  randcnasation  of  the  starting 
configuration  is  relaxed  [21,  after  repeating  the  optimizaUon  wit!  many 
different  initial  flow  configurations,  the  minimum  of  the  local  mtmm  so 
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fcmri  may  be  chosen  as  a  sul«ptimal  solution  fcr  the  minbmm  of  P(F) 
over  F. 

5.  Description  of  the  Flew  Deviation  Method 

Suppose  we  want  to  minimize  a  performance  function  P(F) ,  which 
already  incorporates  the  "penalty  functions"  as  discussed  above  in  observa¬ 
tion  2,  and  we  have  a  feasible,  non-bifurcated  flow  configuration  FQ  (see 

Fig .  4) . 


Figure  4 


With  this  assumption,  in  FQ  the  cormodity  ,  })  flows  fran  N.  to 

N.  on  one  path  only.  We  define  a  flew  deviation  §  around  FQ  as  a 
flow  configuration  in  which  there  are  two  paths  associated  with  each  con. 
nodity  (i,j)  (i.e.  with  each  pair  of  nodes  and  N.) :  one  is  the 
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old  path  tk  jQ,  present  in  Fq,  the  other  is  n  the  new  path  (see 

Fig.  5) .  The  flew  deviation  §  is  obtained  by  inducing  a  circulation  ^ 

of  caimodity  (i,j)  from  N.  to  N .  along  path  tt.  ..  and  from  N.  to 

J  ljx  3 

Na  along  for  all  i  and  j.  If  ♦y  <  ¥  i,j,  then  the 

flew  union  Fq®§  is  also  a  feasible  flew  configuration  (see  Fig.  5). 


Figure  5 

An  infinitesimal  flew  deviation  6$  consists  of  infinitesimal  6<{k  ^ , 

Vi»j. 

Consider  new  the  variation  of  the  performance  function  corresponding 
to  an  infinitesimal  flew  deviation  6§: 

6P(FQ)  =  P(Fq®6$)  -  P(Fq) 
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We  wish  tc  find  <f§  for  which  6P(Fq)  is  minimum,  i.e. ,  the  deviation  of 

g 

maximum  decrease.  But: 


p(F0)  P(fl'^2'^3' 


and 

(4) 


6P(Fq) 


b 

=  E 

k=l 


m 


k  F=F. 


where  b  is  the  number  of  arcs  and  6f^  is  the  infinitesimal  flow  varia¬ 
tion  in  arc  K  induced  by  6$. 

K 

Referring  to  Fig.  5,  let  us  define  the  indicator  functions  .  .  and 


1 

0 


iff  ^ 
130 

otherwise 


includes  arc 


k 


1  iff  tt^ includes  arc  k 
0  otherwise 


Thus  indicates  the  old  routes,  relative  to  FQ  and  {v^}  indicates 
the  new  routes/  created  by  the  deviation  around  Fq .  It  is  easy  to  see  that/ 
in  the  ncn-bi furcated  flow  FQ,  is  given  by: 


^e  recall  that,  without  loss  of  generality,  only  the  minimization 
problen  is  considered. 
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The  variation  6f,  of  the  flow  in  the  "arc  k  after  the  deviation  is  there- 
fore: 

(5)  <Ai  - ««) 

1=1  ]=1  J 

Fran  (4)  and  (5)  we  get: 


For  fixed  step  size  we  may  minimize  6P  by  properly  choosing  the 

new  routes  {v}.  Hence,  our  problem  becomes: 


(5a)  min 
over  {v> 


n  n 

<$p  “  £  £ 

i=l  j=l  13 


nun 

over  {v} 


Y  /A2.  )  L*.  -  £.) 


fbr  each  arc  k,  we  now  introduce  a  key  definition  of  length  as 
follows:^ 


(5b) 


**"  ^F=F, 


Thus,  for  each  (i,  j)  pair  we  have  to  minimize: 


9lOe  case  of  9P/9fk  <  0  (and  therefore  of  ^  <  0)  could  create 

negative  cycles.  The  occurrence  of  negative  cycles,  hcwever,  is  of  no 
iinportance  in  the  practical  case  of  S/F  networks,  sinae  the  delay  is  an 
incr^^ing  function  of  the  flow  in  an  arc  and  therefore  —  (9T/9f^)  >  0. 
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(5c) 


nan 

{v} 


yli±) 

l£i(afk40  1  «  ^ 


min 

{v} 


b  .  • 

E  \  v 

k=l  K' 


k 

ij 


-£.) 


-  (m  lA  -  Ia  ^ 


If  w>  interpret  as  the  length  of  the  arc  k,  then 


iron 


IlkUJ  ^  0  y 

(v)  \  Tc  ij 


is  the  shortest  distance  from  node  i  to  node  j  (i.e.,  the  length  of  the 
shortest  path  connecting  i  to  j)  under  the  metric  I.  The  shortest  path 
(SP)  from  i  tc  j  therefore  defines  the  selected  path  for  deviation  of 

flow  (called  deviation  path) .  ^ 

After  the  deviation  paths  are  found  for  all  pairs  of  nodes,  '  the 

next  step  is  to  determine  the  Wy).  i-«-  =izc  °£  thB  deviation- 
AS%^itta  based  on  the  JD  method  is  outlined  as  follows: 

„  11 

1.  Find  a  feasible  starting  Fq. 

2.  Let  initially  n  =  0. 

3.  Canpute  the  SP  mtx  for  Fn- 

4.  Determine  optima*  deviation  size  and  compute  Fn+1  (see  discussion 

below) . 

5.  If  tP(Fn)  '  P(FntlH  '  E  ”here  £  Is  311  acceptable  threshold 

on  the  performance  improvement ,  let  n  ■*  n  +  1  and  go  to 
nt-hfirwise  stop. 


i«A  very  simple  aigo^ue  toFlcyed.9, .  g-the^hort^  route. 

toto' m  2E*«2  £2*  °£  cJii-  "I*"*  is  *■"  °rder  °£  "  ' 

where  n  is  the  number  of  nodes. 

11See  the  application  section  for  examples. 
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6.  l  imitations  of  the  Method 

a.  Even  starting  with  a  non-bif  urea  ted  flow  Fq,  we  notice  that, 
at  the  end  of  the  first  iteration,  the  flow  from  i  to  j  takes  two 
routes  rr^jQ  and  on  ***  amount  of  ccnmodity  (i,j)  is 

(rij  “  4>ij)/  on  irij1  it  is  (see  Fig,  5). 


Figure  6 

Ac  the  next  iteration,  as  the  flew  configuration  F  has  changed,  also 
the  metric  changes  (see  Eg.  (5b))  and  a  new  shortest  path  tk ^ 
from  i  to  j  is  found  (see  Fig.  6) .  Suppose  tk ^  is  different  frem 
Tr^jQ  and  P^oblan  is  new  hew  to  define  the  FD,  i.e.,  hew  to 

redistribute  the  requirement  r^j  amoung  the  three  paths.  Since  each 
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iteration  adds  a  new  path  (typically)  to  each  pair  of  nodes,  we  realize 
that  a  criterion  for.  this  redistribution  is  needed.  A  reasonable  criterion 
is  the  following:  Suppose  there  are  n  routes  on  which  the  requiranent 
flows  fran  i  to  j .  Choose  the  longest  of  these  n  routes  under  the 
new  metric  and  define  the  deviation  as  a  deviation  of  flew  from  the 
longest  to  the  shortest  path.  Other  criteria  can  be  devised,  but  none  of 

than  (including  this  one)  guarantees  optimality. 

b.  The  existence  of  many  routes  between  each  pair  of  nodes  represents 
a  problem  also  fran  the  point  of  view  of  the  description  of  the  flow.  We 
must  keep  a  record  of  all  the  routes  with  the  associated  weights  (flows 
that  they  carry) ,  and,  if  the  network  is  large,  the  problem  of  storing 
these  data  in  the  computer  during  solution  can  be  severe. 

A  suboptimal  criterion  to  keep  the  number  of  parallel  routes  <_  M 
is  the  following:  If  there  are  m  >  M  routes  from  i  to  j,  eliminate 

those  m  —  M  with  smallest  flews. 

c.  After  having  resolved  points  (a)  and  (b)  with  sane  suboptimal 
criterion,  we  are  a  till  faced  with  choosing  the  deviations  .  In  the 
gradient  method  the  determination  of  the  step  size  is  a  one-dimensional 
optimization  problem.  Here  we  have  n(n  -  1)  variables  ♦y  (where  n 
is  the  number  of  nodes  in  the  network) .  Furthermore,  the  presence  of  the 
penalty  functions  in  P(F)  guarantees  that  the  constraints  are  satisfied 
only  for  a  small  deviation  M.  Large  <jKj  might  produce  a  flow  Fx 


«  F  ®  $ 
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which  is  outside  the  constraint  set.  Thus  the  determination  of  the  optimum 

12 

represents  a  fairly  complex  problem. 

7.  The  FD  Method  for  Large  Networks  with  Balanced  Traffic  Requirement 

It  can  be  shown  that  the  limitations  mentioned  in  the  preceding  section 
have  practically  no  effect  on  large  networks  with  a  balanced  traffic  require¬ 
ment.  The  definitions  of  "large"  network  and  of  "balanced  traffic"  are  more 
qualitative  than  quantitative,  and  they  vary  frcm  problem  to  problem. 

We  begin  by  discussing  the  notion  of  "balanced"  networks.  Let  us  con¬ 
sider  a  network  with  n  nodes.  If  its  associated  graph  is  complete  (i.e., 
all  pairs  of  nodes  are  connected  by  directed  arcs) ,  then  the  nunber  of  arcs 
is  n  (n  -  1) .  Suppose  we  restrict  our  analysis  to  graphs  which  have  Kn 

arcs  with  fixed  K  <  n  -  1;  K  represents  the  average  arc  to  node  density 
13 

of  the  graph. 

£rij 

Let  r  =  _  w  -  n^n  I  iy  be  the  average  flow  requirenent/pair 

of  nodes 


Let  m  =  max  [r.  ./r] 

ij  3 


m  gives  a  measure  of  how  balanced  the  traffic  is.  Notice  that  m  >_  1 
always  and  that  m  =  1  corresponds  to  a  uniform  traffic  requirement  matrix 


l^The  rigorous  approach  to  this  problan  is  to  consider  the  global  flew 
as  a  convex  combination  of  non-bifur cated  flows  and  to  apply  the  method  of 
the  Lagrange  multipliers.  This  approach  is  new  under  investigation  here 
at  the  Computer  Science  Department  of  UCLA. 

13In  many  practical  cases,  the  number  of  arcs  grews  linearly  with  the 
number  of  nodes. 
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(  r.  .  =  r  for  all  i  ^  j) ;  m  is  large  for  very  diversified  require- 

IV 

merits. 

Let  us  now  relate  these  ideas  to  the  notion  of  "large"  networks.  Let 
f  be  the  average  channel  flew  in  the  network;  f  is  given  by: 

7  -  (total  flow  in  the  net)  _  nr(n  -  l)n  _  r(n  -  l)n 
(#  of  arcs)  Kn  K 

where  n  is  the  average  nunber  of  arcs  in  a  path  (typically,  it  is  an 
increasing  function  of  n) ,  and  is  oomonly  referred  to  as  the  average 
path  length. 

Suppose  the  maximum  requir orient  is  routed  on  one  path  only.  Then  the 
ratio  n  between  the  maximum  requirement  and  the  total  average  channel 
flow  is 

mr  mrK  mK 

(6)  n  =  —  - - z  - - z 

f  r(n  -  l)n  (n  -  l)n 

If  we  have  a  "large"  net  with  "balanced"  traffic,  then  n  «  1*  In  this 
situation  one  notices  that: 

flew 

1.  The  ratio  between  each  ccnmodity  /find  the  total  flew  in  a  channel 
is  «  1,  therefore  the  deviation  of  the  commodity  from  the  old  path  to 
the  new  SP  can  be  considered  as  infinitesimal .  Far  this  reason,  limitation 
(c)  mentioned  in  the  preceding  section  can  be  neglected  and  both  improve¬ 
ment  in  P(F)  and  satisfaction  of  the  constraints  are  virtually  guaranteed. 

^  Many  other  appropriate  definitions  of  m  are  possible,  for  example, 
r  /  r  •  .  V 2-il/2 

m*  =  Wl - J  J  ,  in  which  case  m'  =  0  corresponds  to  the  uniform 

traffic  requirement. 
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2.  Again,  as  the  deviation  can  be  considered  as  infinitesimal,  it 
does  not  pay  much  to  split  the  requirement  r^  between  more  than  one 
route  from  i  to  j.  Thus  the  optimum  flew  is  approximately  non-bifur cated 
and,  in  the  application  of  the  algorithm,  we  can  deviate  the  entire  commodity 
from  the  old  path  to  the  new  shortest  path  at  each  iteration.  In  this  way 
the  limitations  (a)  and  (b)  of  the  FD  method  can  be  neglected.  As  most 
of  the  S/F  networks  can  be  classified  as  "large"  and  "balanced,"  we  will 
consider  only  this  category  in  the  applications  section. 

8.  Case  of  P(F)‘  Convex.  The  Routing  Assignment 

Suppose  P  =  P(fi,f2  •  ••  fb)  is  a  convex  function  with  respect  to  each 
of  the  f^s  (see  Fig.  7) :  This  is  the  case  of  Problem  A  (the  routing 
assignment),  mentioned  in  Section  2.  If  P(F)  is  convex,  then  the  equiva¬ 
lent  length  JL  of  the  arc  i,  given  by  3P/3f ^  is  an  increasing  function 
of  fi.  The  fact  that  JL  increases  with  corresponds  to  a  tendency 
of  the  FD  method  towards  spreading  the  flews  in  the  network.  If,  after  an 
iteration  of  the  algorithm,  many  new  routes  include  the  arc  i,  then 
becomes  large,  and  so  does  the  equivalent  length  JL .  This  effect  tends  to 
block  the  further  increase  of  and  to  make  the  other  arcs  appear  more 
favorable  in  the  next  iteration.  From  an  intuitive  point  of  view  one  can 
see  that,  due  to  this  spreading  tendency  the  FD  method  drives  the  flow  to 
a  unique  final  configuration,  no  matter  what  starting  configuration  was 
chosen.1^  This  intuitive  argument  is  substantiated  by  the  mathematical 

1^This  is  rigorously  true  only  if  the  flew  deviations  are  infinitesimal . 

In  the  practical  cases  the  size  of  the  steps  is  finite,  and  the  minimum  flow  can 
be  determined  only  with  an  accuracy  proportional  to  the  step  size.  For  the 
same  reason,  different  starting  configurations  might  produce  different  final  flew 
configurations,  whose  "distance"  from  the  optimal  solution  is  again  propor¬ 
tional  to  the  step  size. 
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argument  that,  in  a  problem  with  convex  performance  function  and  convex 

constraint  set  (both  feasible  and  additional  constraints  define  a  convex 

constraint  set) ,  a  local  minimum  is  also  a  global  minimum.  In  order  to 

understand  how  the  FD  method  operates  in  the  case  of  P(F)  convex,  let 

us  consider  its  application  to  Problem  A.  Let  tu  jQ  be  the  old  route 

and  tn  ^  be  the  new  shortest  route  between  nodes  JL  and  (see 

Fig.  8a),  and  let  a  be  the  amount  of  ccrmodity  (i,j)  deviated  frcm 

a 

^ijO  to  ^iji*  Frcin  (1")  it  is  easy  to  see  that  P(a),/sum  of  convex 

functions  of  a,  is  also  convex  in  a  (see  Fig.  8b) .  The  fact  that 

^i-l  ^  "stottet"  than  it  jQ  corresponds  to  a  negative  slope  at  a  =  0 

(see  Fig.  8b) :  hence  the  FD  method  deviates  an  amount  of  oonmodity 
in  the  proper  direction  14« 

(i,j)/frcm  the  path  it^g  to  the  path  it^. 


14*  t 

The  amount  of  ocnmodity  deviated  depends  on  the  particular  algorithm 
used:  for  example,  if  the  "large  and  balanced  net"  assumption  holds,  then 
a  "ncn-bifurcated"  algorithm  can  be  applied  and  the  total  amount  is  deviated. 
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Then,  the  equivalent  length  =  g-j—  is  decreasing  with  f  and  this 

1 

corresponds  to  a  tendency  of  the  FD  method  to  concentrate  the  flows  in 
the  network .  If,  after  an  iteration  of  'die  algorithm,  many  new  routes 
include  the  arc  i,  then  increases,  JL  decreases  and  thei-.jfcre,  on 

the  next  iteration,  arc  i  will  probably  be  included  in  more  SP's. 

As  it  will  be  shown  in  the  Applications  Section,  Problem  B  (routing 
and  capacities  assignment,  linear  cost-cap  case)  has  P(F)  concave.  In 
that  case  we  know  that  we  can  reduce  the  average  delay  T  by  concentrating 
the  capacities  [1]  ?  so  the  FD  method  actually  works  in  this  direction, 
because  it  concentrates  the  flows,  and  therefore  the  capacities  (see 
Eq.  (3))  .  It  can  further  be  shown  for  Problem  B  that  P(F)  is  also  a 
concave  function  of  a,  where  a  is  the  deviation  applied  to  a  pair  of 
nodes  i,j  (see  Fig.  10): 


Figure  10 
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In  the  case  of  Fig.  10,  if  the  now  was  initially  in  *ij0.  “«*  lf  'te 

SP  moc  tells  us  that  is  shorter  than  ryo  under  the  metr  k> 

tten  the  ID  method  deviates  all  the  flow  fra”  ”i:j0  to 

An  interesting  property  of  the  case  of  POT)  concave  with  respect  to 

«.  now  deviation  a  is  that.  own  wjttout  toe  ..large”  and  glanced” 

.  . _ ,  xn~.  v  ie  not  Hi  furcated .  Let  us  prove 

net  assumption,  the  optimal  flew  F  1  - - 

this  strong  property: 

Suppose  the  optimal  solution  has  two  routes  fran  i  to  D,  "ijl 

.  n  and  n  with  weight  w»  =  aQI  (0  ±  «q  <.  rij^ 

with  weight  wx  =  r..  -  aQ  and  n.j2  2  0 


\ 

i 


Figure  11 


As  we  can  see  from  Fig.  11,  due  to  the  concavity  of  P ( d> ,  the  nummum 

is  on  one  of  the  atxenes  of  the  interval  (O.ry).  and  die  optimal 
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solution  cannot  be  bifurcated,  unless  P  (ot)  is  constant  with  a.  In 
the  latter  case,  any  value  of  a  is  optimal,  extreme  included.  The 
same  argument  can  be  extended  to  the  situation  of  more  than  two  paths 
from  i  to  j,  by  taking  them  two  at  a  time.  We  have  proved,  therefore, 
that  the  optimal  solution  is  never  bifurcated  in  the  case  of  P(F)  conca.a. 

As  far  as  the  "optimality"  of  the  method  is  concerned,  the  starting 
flow  configuration  and  the  order  in  which  we  update  the  flews  strongly 
determine  the  local  optimum  reached,  since  the  more  flew  we  route  on  an 
arc,  the  more  we  would  like  to  route  on  it  at  the  next  iteration.  So, 
the  application  of  the  FD  method  results  in  finding  local  minima,  and  is 
not  as  powerful  here  as  in  the  convex  case.  On  the  other  hand,  the  nan- 
bifurcated  nature  of  the  optimum  solution  greatly  simplifies  the  algorithm 
in  the  cases  where  the  "large"  and  "balanced"  net  assumption  is  not  valid. 

10.  Applications 

As  an  application  of  the  FD  method,  two  multiccmnodity  problems  rela¬ 
tive  to  the  AFPA  Computer  Network  are  presented. 

The  ARPA  Computer  Network  is  a  S/F  communication  network  connecting 
several  computer  facilities  in  the  United  States.  A  detailed  description 
of  the  Network  is  given  in  Refs.  [3-8] .  One  of  the  earlier  proposed 
topologies  of  the  Network  is  given  in  Fig  •  16  •  The  efficiency  of  the 
system  is  measured  in  terms  of  average  delay  T  per  message. 

first  application  is  Problem  A  in  Section  2:  V*e  have  to  find  the 
routing  of  the  flows  in  this  network  which  minimizes  T,  given  the  capaci¬ 
ties.  The  second  application  is  Problem  B:  We  are  given  the  total  cost 
of  the  network  and  we  want  to  determine  the  optimum  routing  and  the  optinum 
assignment  of  the  channel  capacities,  under  linear  cost-cap.  functions. 
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Application  1:  The  routing  assignment 

Fran  Section  2,  the  average  delay  per  message  T  is  given  by: 


T  = 


The  mu! tioamcdity  problem  in  this  case  is  stated  as  follows: 

given:  -  topology  21  nodes  (see  Fig.  16) 

-  assumed  unif .  requirement  mtx:  r^^  —  r  [bits/sec] 

-  capacities  (C^  =  50  kbits/sec, for  all  i) 


minimize, 
over  F 


constraints: 

(i)  F  feasible 
(ii)  f.  <  C. 

Vfe  recognize  that  T  is  in  the  form  of  Problem  A. 

We  first  notice  that:  T(F)  =  <».  So  the  constraint  (ii)  is 

VS. 

contained  in  T(F)  as  a  penalty  function.  Next,  we  want  to  shew  that 
the  "large  and  balanced  network"  condition  holds.  From  Eq.  (6) ,  ry,  the 
ratio  for  the  test  of  "large  and  balanced  net"  condition  is  given  by: 


m(Kn) 

T1  - - Z 

n(n  -  l)n 
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In  the  present  case: 


n  =  21 

n  ^  1  (lower  bound  cn  the  average  path  length) 
nK  =  mirber  of  channels  =  52 
m  =  1  (uniform  requirement) 

H®ce:  n-  (21?W°  °‘12  <<C  1 

and  the  test  is  satisfied. 

The  FD  method  can  therefore  be  applied  in  the  simple,  non-bif urea  ted 
form  corresponding  to  the  "large  and  balanced  net"  situation. 


Figure  12 


Ihe  equivalent  length  JL  (see  Fig.  12).  is  given  by: 


Xt-o 


e.  If  0  >  1,  then  we  have  found  a  feasible  flow.  Go  to 

s 

Phase  2. 

f .  If  0  <  1#  we  cannot  send  the  requirement  r  on  the  routes 

8  n 

we  just  fourxl  without  violating  sane  capacity  constraints.  In 

this  case  a  requirement  of  0.95  0  r,  very  close  to  saturation 

is  then  & 

is  chosen.  The  FD  method  /  applied  to  this  flow,  and  a 
new,  better  routing  is  found  that  raises  the  saturation  level 

to  0;r. 

g.  If  01  =  0  ,  no  improvement  occurred;  0gr  is  the  max  level 

s  s 

of  traffic  that  the  network  can  accept.  Hence  the  problem  is 
infeasible . 

h.  Suppose  0g  >  0g: 

hi:  If  0'  >  1  go  to  Phase  2 

s  step 

h2:  If  0'  <  1  r  update  and  go  to  /f 
s  — 


Phase  2: 

a.  For r^/fSsTcurrently  under  consideration  compute  the  equivalent 
lengths  given  by  Eq.  (7) . 

b.  Compute  SP  matrix. 

16 

c.  Give  flow  deviations  according  to  SP  matrix. 


16jt  pinar  that,  in  order  to  maintain  the  assumption  of  infinitesi¬ 
mal  deviation,  it  would  be  best  to  recompute  the  SP  matrix  after  the  devia¬ 
tion  of  each  single  oaimodity.  However,  the  overhead  due  to  the.reo^^ 
ticn  of  the  SP  matrix  at  each  step  might  be  severe,  so  a  caipraru.se  between 
S^S/Sd  speed  has  to  be  found,  m  cur  escsple,  the  t»  extreme  <*ses 
of  Updating  the  SP  matrix  after  processing  each  carmodity,  and  after  pro 
cessing  all  the  oarmodities,  have  been  investigated:  the  difference  m 
the  final  results  was  less  than  5%,  which  shews  that,  at  least  in  this  p  - 
tim^  example,  the  updating  of  the  SP  matrix  is  not  a  very  critical  factor. 
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d.  If  the  improvement  in  the  performance  functin  T  is  larger 

than  a  predetermined  threshold  e,  go  to  2(a).  Otherwise  stop. 

Figure  17  illustrates  an  application  of  the  algorithm  with  a  uniform 

requirement  r  =  1.2  kbits/sec.  By  routing  the  traffic  according  to  the 

SP  matrix  computed  at  8r  —  0,  saturation  occurs  at  0^r  =  1.02  kbits/ 

sec,  i.e.  at  B  --  .85  (see  curve  a).  As  8  <  we  are  still  in 

Phase  1  and  the  FD  method  is  applied  to  a  flew  corresponding  to  a  reduced 

requir orient  r 1  —  .95  8  r  =  .96  kbits/sec;  new  routes  are  found  (see 

curve  b) ,  and  saturation  occurs  new  at  8^r  —  1.250  kbits/sec,  i.e.,  at 

8'  =  1.04.  As  8'  >  1,  a  feasible  solution  at  r  =  1.2  kbits/sec  is 
s  s 

available,  and  Phase  2  can  be  started.  The  PD  method  is  new  applied 
using  the  full  traffic  requirement  and  the  optimum  routing  is  found  (see 
curve  c) .  Notice  that,  as  expected,  the  routing  (a)  gives  the  best  per¬ 
formance  at  low  traffic  levels,  while  (c)  gives  the  best  performance  at  the 

required  traffic  level. 

This  algorithm  has  been  programmed  in  FORTRAN  and  run  on  the  360/91 
at  UCLA.  The  three  curves  plotted  in  Fig.  17  were  computed  in  8  sec.,  CPU 
time. 

Application  2;  Routing  and  capacities  assignment,  linear  cost-cap,  case 
It  was  shown  in  Section  2  that  this  problem  is  equivalent  to  the 
following  multicommodity  problem: 


given: 


minimize. 
over  F 


-  topology 

-  assumed  uniform  requirement  nvt.< 

b  2 

(f'W)  !  » 

T(F>  =  ^ - +  7  r  i 
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constraints : 


(i)  F  feasible 


(ii)  D  =  D  -  E  >  0 


i-1 

where  di  =  cost/unit  of  capacity  in  channel  i 
D  =  total  number  of  dollars 


We  first  notice  that: 


lim  T(F)  =  00 

D  -»0 
e 
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Note  that: 


lim 


.-K) 

1 


a.(f> 


oo 


due  to  the  term 


1 


It  can/be^roved  that  T  is  concave  with  respect  to  a  flew  devia¬ 
tion  a  [1].  Fran  the  concavity  of  T  we  recognize  that  we  are  in 
the  situation  of  Section  3  and  the  properties  there  mentioned  hold. 

Hence  the  FD  method  can  be  applied  in  the  simple,  non-bi furcated  form. 

As  it  was  pointed  out  in  Section  9,  the  ED  method  leads  to  local  optima, 
and  several  starting  flow  configurations  nust  be  tried.  In  order  to  get 
several  starting  flow  configurations,  the  follcwing  randomization  procedure 
is  repeated  for  each  iteration  of  the  algorithm. 

a.  Assign  initial  equivalent  lengths  \  to  the  channels  at 
random,  so  that  m  <  ^  <  M,  where  m  and  M  are  respec¬ 
tively  lower  and  upper  limits  conveniently  chosen. 

b.  Compute  SP  matrix. 

c.  Route  the  requirements  r  on  the  SP's,  and  get  the  starting 
flow  configuration  Fq. 

d.  If  (D  -  Efjdj)  >  0,  Fq  is  feasible,  the  FD  method  can  be 
applied. 

e.  If  (D  -  £f.d.)  <  0,  Fq  is  unfeasible  and  is  rejected.  Go 
to  (a) . 
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■rne  initial  ranicm  ctoice  of  the  lengths  guarantees  a  certain 
ra*cmness  in  the  stating  feasible  flow,  thus  providing  a  method  for 
finding  several  local  minima.  After  a  convenient  nutter  of  iterations, 
the  global  minimum  is  ctosen  as  the  mininum  of  the  local  minima.  This 

provides  a  "suboptimal"  solution. 

A  block  diagram  of  the  method  is  given  in  Fig.  14. 


Note  1 

m  the  ARPA  network  case,  the  cost^apacity  functions  are  step 
functions;  in  order  to  be  able  to  use  our  mettod,  this 
staircase  function  has  been  approved  by  several  straight  lines,  each 
used  in  the  proper  range  of  capacities  (see  Fig.  15) .  *  way,  giwn 

a  flow  configuration  F,  a  proper  set  of  linear  cost-cap.  functions  is 
chosen:  during  the  application  of  the  FD  method,  this  set  is  updated 
iteration  by  iteration,  as  F  changes.  At  the  eni  of  the  "continuous" 
optimization,  perform*  using  litear  cost-capacity  functions,  the  contmuous 
capacities  are  discretized  to  the  upper  values  of  capacity  available  (in 
the  case  of  Fig.  15,  Cc  *  Cfl) .  Passing  from  the  continuous  to  the  dis¬ 
crete  solution,  the  delay  T  obviously  decreases,  but  the  cost  constraint 
night  he  violated.  In  this  case,  the  "local"  optimal  solution  is  rented, 
or,  tetter,  is  consider*  as  a  l~al  optimal  of  a  different  problem,  there 
the  nether  of  dollars  is  conveniently  larger.  This  technique  is  clearly 
suboptimal,  and  it  is  difficult  to  tell  hew  close  this  solution  is  to 
the  real  optimum.  The  rigorous  approach,  hewerver,  requires  integer  pro- 

i^  es  and  is  very  time  oonsuning.  Other  suboptimal  techniques 
granuung  techniques  ana  is 

have  been  proposed'  /  but' it  is  still  an  open  problem 
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k  =  k  +  1 


no 
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Figure  14 
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Figure  15 


Note  2 

Ws  can  evaluate  the  optimum  configurations  corresponding  to  several 

assignments  of  total  cost  D,  keeping  the  requirement  r  as  a  parameter. 

In  Fig.  18,  several  curves  T  =  T(D)  are  plotted  for  different  values  of 

suboptinun 

r.  Only  the  marked  points  on  these  curves  correspond  to  /  solutions, 

as  the  ensemble  of  capacities  available  is  discrete.  The  set  of  capacities  and 
corresponding  costs  considered  in  the  optimization  can  be  found  in  Table  1 . 

12.  Conclusion 

Fran  the  theoretical  point  of  view,  the  FD  method  is  valid  for  any 
unconstrained  multiocnmodity  problem.  In  the  case  of  constrained  problems, 
one  must  consider  the  trade-off  between  introducing  complicated  penalty 
functions'  and  being  able  to  use  an  algorithm  generally  easier  than  the 
constrained  optimization  algorithms,  i.e.  linear  or  non-linear  programming. 
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table  1 

CHANNEL  CAPACITIES  AND  CORRESPONDING 
COSTS  USED  IN  THE  OPTIMIZATION 

Capacity 
[kbits/sec] 

7.2 
19.2 
50 
108 
230.4 


Note:  The  total  cost  per  month  of  a  channel  is  given  by: 

total  cost  =  termination  cost  +  (Hne  cost)  X  (length  in  miles) 


Termination  Cost 
f$/ftnonth] 


810 

850 

850 

2400 

1300 


Line  Cost 
[$/month/mile] 


.35 

2.10 

4.20 

4.20 

21.00 
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If  the  problem  automatically  incorporates  the  penalty  functions  in  the 
objective  function  (as  S/F  network  problems  usually  do) ,  and  if  the  "large 
and  balanced  net"  assumption  is  valid,  then  the  FD  method  seems  to  be 
more  convenient  than  other  methods. 
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AN  ANALYSIS  OF  THE  SEPARATION  BETWEEN  JACKETS 
IN  A  STORE-AND-FORWARD  NETWORK* 

by 

G.D.  COLE 
and 

L.  KLEINROCK 

Computer  Science  Department 
University  of  California 
Los  Angeles,  California 


ABSTRACT 

The  ARPA  computer  network  utilizes  message-switching  to  provide  communications 
between  the  various  HOST  computer  sites.  This  intercommunication  is  in  terms 
of  messages,  which  are  segmented  into  smaller  size  packets  for  store-and-for- 
»#srd  transmission  between  the  message-switching  computers  which  are  referred  to 
as  Interface  Message  Processors  (IMP’s).  These  segmented  (multi-packet)  messages 
must  be  reassembled  at  their  destination  IMP  prior  to  delivery.  Each  IMP  receives 
packets  on  two  or  more  input  lines,  causing  the  occasional  buildup  of  queues  on 
a  given  output  line.  The  result  of  such  queues  is  that  the  packets  of  a  multi¬ 
packet  message  can  become  separated,  i.e.,  "foreign"  packets  can  become  inter¬ 
spersed  with  those  of  a  given  message.  This  separation  causes  a  longer  reassembly 
time  at  the  destination  IMP,  and  is  therefore  of  interest  from  the  viewpoints  of 
message  delay  and  buffer  utilization. 


The  analysis  of  the  packet  separation  assumes  Poisson  arrivals  of  interfering 
traffic,  and  considers  both  first-come,  first-served  and  priority  queue  disciplines 
at  the  output  transmission  facility.  Equations  are  derived  for  the  expected  inter¬ 
packet  separation  in  each  case,  and  both  expected  values  are  shown  to  have  the 
same  finite  limiting  value  as  the  packets  pass  through  an  arbitrarily  large  number 
of  nodes.  Empirical  results  are  also  given  based  on  measurement  data  taken  on  the 

ARPA  net,  and  the  relationship  between  the  modeling  and  measurement  efforts  is 
discussed. 


*  This  research  was  supported  by  the  Advanced  Research  Projects  Agency 
Department  of  Defense  under  Contract  Number  DAHC-15-69-C-0285. 
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1.  INTRODUCTION 

The  ARPA  network  of  computers  Involves  an  extensive  store-and-forward  message 
switching  network  to  provide  data  connunicatlons  between  the  various  HOST  sites. 
This  net  utilizes  Interface  Message  Processors  (IMP's)  at  each  store-and-forward 
node  and  SOK  bit/second  transmission  lines  to  provide  both  a  responsive  data 
path  for  interactive  work  and  a  reasonable  throughput  for  file  transfers. ^  ^ 

The  HOST  sites  consist  of  a  variety  of  computer  systems,  with  these  hardware  and 
software  resources  being  shareable  via  the  network.  Several  of  the  sites  have 
developed  particular  specialties  such  as  graphics,  artiflcal  intelligence,  man- 
machine  interaction,  etc.  UCLA  is  one  such  node,  having  specialized  in  the  areas 
of  network  performance  analysis,  simulation,  and  measurement,  and  serves  as  the 
Network  Measurement  Center  (NMC).  An  extensive  set  of  measurement  tools  has 
been  developed,  both  within  the  IMP's  and  at  the  NMC  HOST  computer.**  These 
facilities  provide  an  experimenter  with  "probes"  to  monitor  the  network  traffic, 
or  if  desired,  to  create  artificial  traffic  to  simulate  a  given  message  traffic 
condition.  These  measurement  tools  have  been  found  to  be  very  effective  in  gaining 
insight  into  the  network  behavior,  and  in  verifying  (or  correcting)  the  accuracy 
of  analytic  models  of  network  performance.  This  paper  considers  one  such  model, 
namely  the  analysis  of  the  separation  between  message  segments  when  other  traffic 
is  In  contention  for  the  communication  facilities,  and  discusses  the  usage  of  the 
measurement  tools  in  this  development. 

A  message  can  be  at  most  8064  bits  in  length  which,  at  a  50K  bit/second  rate, 
requires  about  160  milliseconds  for  serial  transmission  of  a  full  message  over 
one  of  the  transmission  lines.  For  a  variety  of  reasons*,  messages  are  segmented 
into  shorter  units  (called  packets)  of  up  to  1008  bits  each.  This  segmentation  is 
performed  by  the  source  IMP,  with  reassembly  of  the  packets  into  a  complete  message 
being  handled  by  the  destination  IMP;  Thus  the  HOST-to-HOST  transmissions  are 


*  Message  segmentation  into  shorter  blocks  is  desirable  for  several  reasons 

including  efficiency  of  buffer  allocation  and  error  control,  as  well  as  reduction 
of  retransmission  delays  and  delays  to  nhort  interactive  messages. 
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In  terms  of  messages,  and  IMP-to-IMP  transmissions  Involve  packets.  When  an  IMP 
receives  and  accepts  a  packet  from  a  neighboring  IMP,  It  sends  an  acknowledgement 
(ACK)  In  reply  to  this  nelghiWir  Indicating  that  the  packet  was  acceptable;  this 
allows  the  previous  IMP  to  free  the  storage  buffer  being  utilized  for  that  packet. 


Each  IMP  has  two  or  more  Input  channels  upon  which  packets  can  arrive.  (An 
arrival  Is  said  to  occur  when  the  last  bit  of  the  packet  has  been  received . ) 
Therefore,  It  Is  possible  for  Incoming  packets  tc  arrive  more  rapidly  than  they 
can  be  forwarded,  and  queues  can  build  up  with  packets  from  two  or  more  messages 
becoming  Interspersed  as  shown  In  Figure  1.  The  effect  of  this  lnterspersal  Is  to 
cause  the  packets  of  any  given  message  to  become  separated,  and  therefore  Introduces 
an  additional  delay  In  the  reassembly  of  the  message. 


Output 

Channels 

Completion  of  PICT  1 


Outputs  on  channel  1 


Inputs  on  channel  k 


Inputs  on  Channel 


> 

pkt  i  VmM 

PKT  2 

“1 

PKT  1  starts  serv 

1 

Ice  (Separated  upon 

departure) 

queue  delay 
-•  -  for  PKT  2 

f  Arrival 
^  of  PKT  1 

h - - 

PICT  1 


PKT  2 


PKT  3 


(contiguous  upon  arrival) 


Arrival  Arrival  Arrival 
of  PICT  1  of  PICT  2  of  PICT  3 


Figure  1,  The  Interspersing  of  Message  Packets  at  a 


Store-and-Forvard  Node 
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Theorem  1 

Utilizing  a  FCFS  queue  discipline, 


the  expected  interpacket  gap  for  a 


segmented  message  is: 
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where  t  is  the  interpacket  spacing  due  to  interfering  traffic  at  the 
output  of  the  n  node  along  a  store-and-forward  path;  Xfl,  is  the  service 
time  of  a  maximal  length  packet;  xi  is  the  traffic  intensity  in 

terms  of  the  average  arrival  rate  of  interfering  packets,  and  their 
average  service  requirement,  and  y  =  pt  +  a  where  a  is  the  probability 
that  an  interference  packet  (on  input)  takes  the  same  output  channel  as 
the  observed  traffic.  All  pi's  are  assumed  to  be  equal  along  the  entire 
network  path.  Interfering  packet  arrivals  at  each  node  are  assumed  to  be 
Poisson. 

Proof : 

If  we  consider  a  "tagged"  multipacket  message  at  the  originating  node,  consisting 
of  packets  P^Pj,. . .  ,P. ,. . . ,  we  find  a  situation  as  shown  in  Figure  2.  The 
packets  are  presented  to  the  queue  as  a  group,  and  for  the  FCFS  system,  are 
transmitted  continguously.  The  arrivals  at  the  second  node  are  therefore  spaced 
at  regular  intervals  of  ^  -  Xa  seconds,  where  Xg  is  the  time  required  to  service 
one  maximal  length  packet,  i.e.,  approximately  20  msec,  for  a  1008  bit  packet  at 
50Kb /sec.  However,  at  this  second  node  there  may  be  interfering  traffic  arriving 
on  other  input  channels  (with  an  average  arrival  rate  of  and  as  shown  in 
Figure  2(b),  these  packets  may  become  interspersed  with  the  packets  of  the  tagged 
message .  The  gap  time  (t2)  between  the  now  non-contiguous  packets  has  an  expected 
value  given  by: 

eIt21  *  ni2  xi 

where  the  subscripts  i  refer  to  the  interfering  traffic  and  the  subscript  2 
indicates  that  this  delay  is  at  the  output  of  the  second  node  along  the  path. 
ni2  is  the  average  number  of  interfering  packets  which  arrive  during  the  inter¬ 
arrival  time  between  the  tagged  packets;  for  Poisson  interference  arrivals  this  is: 


ni2  ■  *1  Xs 


(4) 


departure  of  the  jth  packet 


Figure  2,  The  Effect  of  Interfering  Traffic  to  Create  a  Separation  Between  Arrivals 

of  Packets  of  a  Message 
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Equation  (3)  then  becomes. 


E[t2]  -  (Xt  Xg)  xif 


Defining: 


P1  “  Xi  V 


we  have. 


e[t2]  -  Pt  Xs 


This  separation,  t~,  then  contributes  to  the  output  Interval  at  the  second  node 


such  that: 


E[T2J  -  Xg  +  E[t2],  (  ' 

which  becomes  the  average  lnterarrlval  time  at  the  third  store-and-forward  node. 
There  is  a  non-zero  probability  that  the  interfering  packets  which  separate  the 
tagged  packets  will  be  routed  to  the  same  output  channel  at  this  node, 

a  =  P[ Interfering  packet  stays  as  interference  on  output]  (8) 

The  resulting  gap  between  the  tagged  packets  at  the  output  of  the  third  node 
then  becomes: 

E[t3]  =  X1(Xg  +  EIt2])x±  +  a  EIt2] 
or  substituting  for  E[t21,  and  simplifying. 


E[t3]  -  Pt  Xs  (1  +  y) 


where : 


Y  =  Pj  +  a 
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A  similar  Use  of  reasoning  at  the  fourth  node  produces, 

E[r  ]  -  P1(X8  +  E[t3l)  +  n  E|Tj] 


or: 


e[t4]  -  Pl  x8  [1  +  Y  +  Y  1 

and  leads  to  the  general  iterative  expression: 

E[to1  =  Pl(X8  +  ElVl1)  +  °  ElVl] 


(10) 


or: 


n-2 

EI'J  -  ’i  *s  £  (,)J 

3*0 

Equation  (11)  can  be  expressed  in  closed  form  as: 


-1  Pi 

ElxJ  -  U  -  Mn"  1 


(ID 


(12) 


*ith  x  being  the  gap  between  a  pair  of  tagged  packets  at  the  output  of  the  nth 
store-and-forward  node.  This  completes  the  proof  of  Theorem  1. 

,  in  Fieure  3  for  several  values  of  Pt,  and 

The  expected  interpacket  gap  s  ,  <>  relatively  small 

indicates  several  interesting  aspects  about  the  gap  size.  For  relat 

lues  of  pt.  the  expected  gap  length  reaches  an  as^totlc  ^  ‘ 

1  .  lflrBpr  values  of  p4  requiring  long  paths  (in  excess 

rhrnuph  a  few  nodes,  with  larger  values  ^ 

through  a  tew  each  ca8e  fln  aaymptote 

°f  ten,r  r:i:r  irrr ;  z  -  -  does  not  m  .  u„b„u„ded 

u -  - — — 


lim  E t Tn 3  =  Too  1  -  y  Xs 
n-*® 


(13) 


V  Thls  fo^TTs  believed  to  be  analogous  to  conservation  results  for  various  time¬ 
sharing  queue  disciplines  (Ref.  7). 


Expected  Inter-Packet  Gap,  t 

(normalized  with  respect  to  the  full  jacket  service  time) 


>  «  O.fl 


y  “  &  +  a 


Y  =  0.6 


Y  =  0.4 


Y  =  0.2 


0  1234567 


8  9 


Number  of  Nodes  Along  the  Store-and-Forward  Path,  n 


FIGURE  3,  THE  EXPECTED  INTER-PACKET  SEPARATION 
GOING  THROUGH  n  NODES 
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It  is  also  Interesting  to  note  that  for  the  expected  interpacket  gap  to  become 
within  a  factor,  F,  of  the  asymptotic  value,  t^,  we  need: 

„  s  i  +  is&a  -  (i4) 

log  Y 

This  can  be  shown  by  noting  that  the  expression  for  E [ ]  can  be  written  as: 

E[tn]  -  [1  -  (y)0'1]  7.  -  F  7. 
such  that: 


(y)""1  -  1  -  F 

and  by  taking  logarithms  of  both  sides, 

n.1  +  i2sii^a 

log  Y 


Therefore,  an}'  v^lue  of  n  equal  to  or  greater  than  this  value  will  cause  the 
expected  gap  to  be  within  the  factor,  F,  of  the  asymptotic  delay  value. 


A  variation  of  interpacket  gap  model  was  obtained  by  considering  the  case  of  non- 
uniform  traffic  intensities  along  the  store-and-forward  path.*  This  change  meant 
that  the  expected  interpacket  delay  would  not  necessarily  increase  at  each  sub¬ 
sequent  node,  resulting  in  a  variation  of  the  iterative  relationship  of  Equation 
(10)  to  Include  this  case. 


E[tb]  -  “*<Pln(xs  +  EtTn-l]  ’ElVl])  (15> 

where  p.  is  the  traffic  intensity  o£  the  interfering  packets  at  the  nth  node. 

In 

(This  previous  analysis  was  for  the  a-0  case.) 

*  This  extension  was  made  by  Gary  Fultz,  working  under  the  supervision  of  one  of 
the  authors  (LK),  who  was  concerned  with  differing  traffic  intensities  across 
the  net  in  routing  simulation  studies.  This  interpacket  gap  time  was  of  concern 
to  him  because  of  its  effect  on  the  reassembly  time  of  messages  at  the  eventual 
destination.  (See  Ref.  8.) 


1  'I'P 

-LO/v, 

il 


3.  THE  INTRODUCTION  OF  PRIORITIES 

The  interpacket  gap  analysis  in  a  priority  system  differs  in  two  aspects  from  that 
of  the  FCFS  system.  In  the  FCFS  system,  any  arrivals  occurring  between  the  arrivals 
of  packets  and  P^+^  were  also  serviced  between  P^  and  the  priority 

system,  the  interfering  arrivals  will  be  any  non-priority  packets  between  the 
arrivals  of  Pj  and  and  any  priority  arrivals  after  P^  begins  service  but 

before  P^+^  begins.  The  resulting  interpacket  gap  given  in  Theorem  2  is  shown 
to  increase  more  rapidly  than  that  of  the  non-priority  case,  but  to  have  the 
same  limiting  value. 


Theorem  2 

The  expected  interpacket  gap  for  a  segmented  message  with  higher  priority 
traffic  competing  for  the  server  is: 


B[tJ 


2,3, 


•  •  • 


(16) 


where  the  conditions  and  variables  are  as  defined  in  Theorem  1  with 
Pp  and  p  being  the  traffic  intensities  of  priority  and  non-priority 
traffic  respectively,  and  p  =  p^  +  p^.  The  value  K  is. 


K 


+  ap 

_ _R 

+  a 


Proof: 

Assume  that  a  multipacket  message  is  initially  entered  into  the  network  as  shown 
in  Figure  4(a).  The  and  j+lSt  packets  of  the  message  are  not  necessarily 
served  contiguously  (as  in  the  FCFS  case)  since  priority  arrivals  during  the 
service  of  P^  will  be  serviced  before  P^+^  as  will  other  priority  messages  which 
arrive  during  the  service  of  the  priority  messages.  The  expected  time  between 
the  departures  of  Pj  and  P^+^  is: 


EfTj]  -  Xg  +  E[Tl] 


(17) 


133 


Figure  4,  Arrivals  and  Departures  for  the  Queue  and  Server  in  the  Priority  Queueing  Case 


(18) 
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where  ^  Is  the  time  to  service  the  priority  arrivals,  with: 
E[vl>  -  Xlp(X8  +  Eltj]), 

*ip  the  average  arrival  rate  of  interfering  priority  meeaagea 

Since  the  arrivals  and  service  requirements  are  independent, 

E[t^]  «  E[v^]  .  E[x  jmes.  is  priority] 


or: 


EtxlJ  *  l * 3ip«.  +  ElfiDSp 
Solving  for  we  obtain: 

rf-  1  _  X.  X.  X  p  X 

■  ip  1  8  *  p  S 

1  -  X.  x  1  -  p 
ip  p  ^1 

where  pp  is  the  traffic  intensity  for  interfering  priority  messages. 

p  =  X.  x 
P  ip  P 

The  expected  interdeparture  time  of  packets  Pj  and  P  at  the  first  node  is  then 


(19) 


(20) 


EfV  -  X  +  Pp  Xs  -  X  +  E[x  ] 

3  1  -  p  8  1 

P 


(21) 


so  that  at  the  second  node  we  can  expect  to  find  n2  interfering  arrivals  of 
regular  messages,  where: 


Eln2] 


Xinp 


Xinp(Xs  +  *'1" 


(22) 
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Only  the  non-priority  -e.a.ge  arrival.  during  T,  .re  considered  to  be  interfering 
in  the  sen.e  of  adding  to  the  interpret  gap.  Any  priority  .rriv.1.  .re  either 

an  Pj  ”  7  COMldered  ln  the  V2  interfering  priority  arrivals, 
wnere  v2  has  an  expected  value  of: 


EtV2>  '  »lp«.  +  «t2]> 

Therefore,  we  have: 

E[T2]  -  Xg  +  E[n2]  +  B[vJ 


np 


xp  +  aEfT1J 


(23) 


(24) 


her.  the  .EfiJ  ter.  1.  due  to  the  frecttou,  ,,  of  the  incoudn,  interfereoc. 
thet  is  routed  .long  .ith  the  tegged  «...g.  (..  defined  l0  e,Mtl01)  „  . 
Substituting  for  £[„,)  sud  E[v2>  fro.  e,uetio„.  (22)  ^  (23)  rMp.etl„ly 
and  noting  that:  y* 

E[t2]  -  e[T2]  -  xg 

we  have: 


E[T2]  "  Pnp  (Xs  +  E*Tl)>  +  Pp  <X8  +  E[t2J)  +  aEftj] 
Collecting  terms  and  simplifying  results  in: 


(25) 


E[t23  "  {p  Xs  +  (pnp  +  a)  EfTi3> 

which  generalizes  into  the  iterative  result: 

ElTnJ  *  rrr-  *,  +  <enp  +  o)  E[r  ])  u  .  2,  3.  ... 


(26) 


(27) 


*  The  effect  of  the  concurrently  routed  interference 


in  the  priority  case,  but  on  the 

component  of  oETt  1. 

in 


traffic  is  more  complicated 


average,  will  produce  an  interference 
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A  closed  fora  representation  of  this  equation  is  difficult  to  find  >y  inspection 
of  the  forms  of  E(t2),  EItj],  etc.,  but  is  expected  from  previous  work  to  be  of 
the  form: 


This  can  be  shown  to  be  correct  by  an  inductive  proof,  i.e.,  utilizing 
equations  (27)  and  (28),  such  that  we  test  for  the  equivalence  of: 


E[tnl  ■  - 


K  +  (P"P  +  0> 


II 


(29) 


Simplification  of  the  left-hand  side  shows  that  the  equality  holds.  We  can  solve 
for  the  value,  K,  by  evaluating  the  expression  for  a  known  condition,  namely  E(t21* 


'-rb-  jpx. +  (pn, +  (r?yj  ■ 

After  considerable  simplification  and  combination 
shown  to  be,* 


of  terms,  the  value  of  K  can  be 


P  +  ap 
*np  MP 

p  +  a 
np 


which  completes  the  proof  of  Theorem  2. 


(31) 


*  These  simplifications  in  volve  the  usage  of  the  equalities  p  -  pp  +  Pnp  and 
y  ■  p  +  a. 
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The  expected  gap  site  for  the  priority  caae  can  then  be  expressed  as: 


n-2 , 3 • • • • 


which  for  p  equal  to  zero,  becomes  the  FCFS  result  of  equation  (12). 

P 

If  we  compare  the  FCFS  and  priority  analysis  results ,  we  see  that  they 
same  limiting  values,  i.e., 

lim  E[t  IFCFS]  -  lim  Eft  |priority  case] 
n 1  n 

n-*«  n-Hi 

This  latter  limit  can  be  found  by  showing  that: 
n 


(P  +  <*\ 

r?T-  •» 

P 


or  equivalently,  that 


I  -  P„  >  Pnn  +  a 
P  np 


Ue  know  that. 


1  >  v  •  p  +p  +  o 
P  np 


and  therefore  by  subtracting  pp  from  each  side  of  the  equation. 


1  "  P„  >  P„„  +  ° 
P  np 


(32) 


have  the 

(33) 


(34) 


which  produces  the  limit  as  in  equation  (34).  This  limiting  effect  is  also 
indicated  in  Figure  5,  which  shows  both  the  priority  and  FCFS  cases. 


Number  of  Nodes  Along  the  Store-and-Forward  Path,  n 


FIGURE  5,  THE  EXPECTED  INTERPACKET  GAP  FOR  THE 
PRIORITY  SYSTEM  AFTER  GOING  THROUGH  n  NODES 
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4.  THE  ANALYSIS  FOR  NON-BULK  ARRIVALS 

The  preceding  analysis  was  based  on  the  assumption  that  all  of  the  packets  of  a 
message  arrive  at  the  source  IMP  simultaneously,  i.e.,  as  a  bulk  arrival.  If  we 
assign  a  finite  rate  to  these  HOST-to-IMP  transfers,  such  that  a  full  packet 
requires  X^  seconds  to  cross  this  boundary,  then  the  analysis  must  be  modified 
to  include  this  effect.  The  Interpacket  gap  at  the  output  of  the  originating  node 
then  has  an  additional  term: 


Ft'ii  ■  *iP  yx„ +  +  »i„P  Vi 


05) 


which  is  a  modification  to  Equation  (19)  to  include  the  non-priority  interference. 
Solving  for  the  average  gap  time,  we  obtain: 


EltJ 


p  X  +  p  X, 
vp  a  vnp  1 

1  -  P„ 


(36) 


The  delays  at  subsequent  nodes  can  be  determined  by  the  previous  iterative 
equation  (27),  with  the  result  being  an  additional  term  beyond  that  of  equation  (32), 


E[t„] 


<0__  +  a) 


n-1 


np  np 


(1  - 


pp)n 


n  =  2,3,  ...  (37) 


where  K  is  as  defined  in  equation  (31),  and  the  additive  term  is  the  effect  of  the 
time  required  for  a  HOST-to-IMP  packet  transmission,  X^  However,  it  is  interesting 
to  note  that  this  additive  effect  does  not  change  the  asymptotic  values  as  n  becomes 
very  large,  since  the  term  of  equation  (37)  approaches  zero  as  n-*«. 

5.  EXPERIMENTAL  VERIFICATION  OF  THE  PACKET  SEPARATION  MODEL 

A  complete  experimental  verification  of  the  packet  separation  model  would  require 
the  ability  to  generate  artificial  traffic  at  each  node  along  the  store-and-forward 
path  of  a  test  message.  Since  such  facilities  are  currently  available  at  only  one 
site,  the  UCLA  Network  Measurement  Center,  the  verification  test  was  much  less 
extensive  than  would  normally  be  desired,  and  was  conducted  only  for  the  case 


19 


of  n  -  1.  Equation  (37)  is  valid  only 
equation  (36) , 
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n  2,  so  the 


test  was  based  on 


P  X 

eiv 


+  p  X. 
P-E-l 

PP 


The  interference  traffic  was  generated  by  the  pseudo-random  artificial  traffic 
generator  (approximately  an  exponential  distribution  of  message  lengths  and  Poisson 
arrivals)  with  the  traffic  intensity,  p,  being  determined  by  selection  of  the 
average  message  length.  The  number  of  active  message  generators  was  a  compromise 
between,  (1)  the  desire  to  avoid  alternate  routing,  and  (2)  the  desire  to  minimize 
the  RFNM  dependency  of  the  arrivals.*  Eight  generators  were  found  to  meet  both 
desires  satisfactorily. 

The  generation  of  the  multi-packet  test  traffic  was  planned  utilizing  two  different 
techniques  to  obtain  a  range  of  values  of  the  parameter,  X^,  (the  time  required  for 
the  serial  "arrival"  of  a  packet).  The  "fake  HOST"  pseudo-message  generator  facility 
within  the  IMP  was  selected  to  provide  a  minimal  value  of  X^  of  3.3  msec.,  which 
is  the  typical  time  required  for  the  background  program  to  generate  a  full  packet 
of  data.  Similarly,  the  "fake  HOST"  generated  traffic  at  a  neighboring  IMP  was 
utilized  for  the  maximal  value  of  X^  •  23.0  msec,  which  is  the  time  required  for 
the  sorial  arrival  of  a  full  packet  at  the  UCLA  IMP. 


The  theoretical  values**  of  the  inter-packet  gap  time,  x,  for  various  message 
lengths  and  values  of  Xx  are  plotted  in  Figure  6  along  with  the  test  data  from  the 
experiment.  The  increase  in  the  value  of  x  for  very  short  messages  is  due  to  the 

*  The  RFNM  (Request  For  Next  Message)  is  a  flow  control  mechanism  which  does  not 
allow  the  next  message  to  be  sent  on  a  given  logical  link,  until  a  request  is 
received  in  reply  to  the  present  message. 

**  See  the  appendix  for  the  calculation  of  these  values . 


Figure  6,  A  Comparison  of  Measured  and  Theoretical 
Inter -Packet  Gap  Times  Due  to  Interference  Traffic 
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fact  that  most  of  these  messages  are  of  the  priority  class,  and  therefore  cause 
an  Increased  amount  of  Interference.  Equation  (37)  Indicates  that  for  the  special 
case  in  which  all  of  the  interference  traffic  is  of  the  priority  class,  the  value 
of  t  will  not  be  a  function  of  X^,  and  this  effect  was  also  verified  by  the 
experimental  data.  This  effect  is  due  to  the  fact  that  the  only  arrivals  that 
will  contribute  to  the  inter-packet  gap  are  those  that  occur  during  the  Xg  seconds 
that  a  packet  requires  for  service,  and  the  subsequent  arrivals  during  their 
service,  etc.,  as  expressed  in  equation  (19). 

The  experimental  and  theoretical  results  agree  reasonably  well  for  both  cases. 

Both  the  shape  of  the  curves  and  the  magnitude  of  the  interpacket  gap  (as  measured 
by  trace  data  at  the  destination)  indicate  that  the  model  represents  the  actual 
system  system  behavior  satisfactorily.* 

This  agreement  between  the  experimental  and  theoretical  results  was  not  obtained 
in  the  initial  experiments,  but  came  about  as  a  result  of  several  improvements 
to  the  original  model.  These  improvements  were  made  as  a  result  of  new  insight 
gained  from  the  measurement  work,  and  included  the  correction  of  an  algebraic 
error,  as  well  as  correcting  a  more  fundamental  oversight  of  having  neglected  the 
effect  of  acknowledgement  traffic  (as  described  in  the  Appendix).  The  measurements 
were  thereby  found  to  significantly  aid  in  the  development  and  verification  of  the 
theoretical  model,  and  conversely,  the  modeling  studies  were  of  great  value  in 
determining  which  system  values  were  of  interest  in  the  measurement  efforts.  The 
two  areas  of  investigation  are  quite  complementary  and  can  be  utilized  together 
to  gain  insight  and  confidence  in  the  correspondence  between  the  behavior  of  models 
and  the  actual  system  being  studied. 


*  The  ability  to  trace  messages  is  one  of  the  set  of  measurement  facilities  at  the 
NMC.  The  trace  data  includes  the  time  of  arrival  of  each  packet,  from  which  the 
value  of  T  can  be  computed  (see  reference  6) . 
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APPENDIX  —  THE  THEORETICAL  TRAFFIC  INTENSITY  INCLUDING 
THE  EFFECTS  OF  ACKNOWLEDGEMENTS  AND  PRIORITY 
TRAFFIC 


The  theoretical  curves  of  Figure  6  include  the  effects  of  the  acknowledgements, 
the  priority  messages,  and  the  non-priority  messages  by  considering  the 
expected  service  time  to  be: 

E[x]  -  E[x  |  an  ACK]  .  Prtan  ACK] 

+  E[x  |  a  pri.  mes.]  .  Prfa  pri.  mes.] 

+  E[x  |  non-pri.  mes.]  .  Pr[non-pri.  mes.] 

The  acknowledgements  are  in  response  to  the  RFNM's  (Request  For  Next  Message), 
and  therefore  occur  with  the  same  frequency  as  messages,  such  that: 


Pr[an  ACK]  *  0.5 


Messages  are  considered  to  be  in  the  priority  class  if  they  are  less  than  or  equal 
to  10  bytes  (80  bits)  in  length,  which  for  the  exponential  density  of  message 
length,  gives  the  following  expression  for  8,  the  fraction  of  messages  that  are 
in  the  priority  class. 


where  l  is  the  average  message  length  in  bytes.  (The  ACK's  will  also  be  in  the 
priority  class).  The  expected  service  time  for  a  priority  message  can  be  found 
from  the  truncated  exponential  density  such  that. 


1 


<*r/8>  «  Xt/;' 

1  -  i  ^/X  . 


x  ■  E [ x  | x  S  X_]  -  X 

p  m  u1.  j. 
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and  the  expected  service  time  for  a  non-priority  message  can  be  shown  to  be: 

x  =  E[x  lx  >  XL]  -  X_  +  X 
np  m  1  m  T  T 

Using  these  results,  one  can  write  the  expected  service  time  (of  messages 
and  ACK's)  as: 

E[x]  =  0.5Xa  +  0.5  0  [XQ  +  xp]  +  0.5(1  -  3  )  [XQ  +  XJ  +  X] 

where  the  first  two  terms  represent  the  priority  service  times,  and  the  last 
term  considers  the  non-priority  service  times.  (The  value  XQ  is  the  fixed 
overhead  service  time  of  a  message.)  The  equation  is  of  the  form: 

E[x]  =  E[x  |  priority]  +  (1  -  B)E[x  |  non-priority] 

and  if  multiplied  by  the  net  arrival  rate  (of  messages  and  ACK's),  produces  a 
traffic  intensity,  p,  given  by: 

« 

p  =  XE[x]  -  X{0.5Xfl  +  0.53  [XQ  +  xp]}  +  X{0.5(1  -  p  ) [XQ  +  XJ  +  X] 
where : 

X  =  X  +  X  -  2X 
a  m  m 

with  the  subscripts  referring  to  ACK's  and  messages.  The  values  of  the  priority 
and  non-priority  interference  traffic  intensities  can  then  be  written  as: 

pn  =  2X  (0. 5X  +  0.5  B  [Xn  +  xl} 
p  m  a  up 

and: 

%  ■  V°-5<1  -  emo  +  Xj  +  xn 
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with  the  composite  traffic  intensity  being: 


If  we  consider  an  example  in  which  the  messages  have  an  average  length  of  20 
bytes,  we  find  that  40%  of  the  messages  are  in  the  priority  class,  and  such 
messages  have  an  average  service  time  of  3.44  msec.*,  compared  to  an  average  of 
7.5  msec,  for  the  non-priority  messages  (including  the  overhead  characters  for 
each).  The  service  time  of  an  acknowledgement  is  3.0  msec.,  and  when  combined 
with  the  priority  messages,  produces  a  traffic  intensity  of: 

p  =  X  [3.0  +  0.4(3.44))  -  0.175 
p  m 

where  \  is  1/(25  msec.)  for  the  artificial  traffic  being  generated.  The  traffic 
m 

intensity  for  the  non-priority  traffic  can  be  found  in  a  similar  manner,  since: 


p  -  X((l  -  0  )[Xn  +  JL,  +  X)} 
np  m  Or 

which  becomes : 

p  -  [2.9  +  1.6  +  3.2  -  0.185 

np  25 

such  that: 


P 


PP  + 


np 


0.36 


The  expected  value  of  t  can  then  be  calculated  for  the  known  value  of  Xg  equal  to 
23.0  msec.,  and  for  the  desired  value  of  X^  These  results  are  tabulated  in  Table 
1  along  with  the  associated  values  of  pp  and  pnp.  (These  are  the  values  utilized 
for  the  theoretical  curves  of  Figure  6.) 


*  Based  on  a  20  ysec./blt  transmission  delay. 
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TABLE  1 

THEORETICAL  VALUES  OF  THE  INTER-PACKET  GAP,  T 


Avg.  Mes. 

Length 

(bytes) 

Expected  Value  of  t 

P 

8 

PP 

pnp 

■pci 

KH 

X1-23.0 

0 

.23 

1.00 

.23 

.00 

6.9 

6.9 

8 

.28 

.72 

.22 

.06 

6.7 

8.3 

20 

.36 

.39 

.18 

.18 

5.8 

10.1 

40 

.49 

.22 

.15 

.34 

5.3 

13.3 

60 

.63 

.15 

.14 

.49 

5.6 

16.8 

80 

.77 

.12 

.14 

.63 

6.0 

20.4 
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Abstract 


An  extensive  measurement  capability  has  been 
Implemented  in  the  Interface  Message  Processors 
(IMP's)  of  the  ARPA  network  of  computers.  These 
measurement  facilities  were  implemented  by  the 
network  contractor.  Bolt  Seranek  and  Newman,  Inc., 
to  meet  the  measurement  needs  of  .the  UCLA  Network 
Measurement  Center,  and  consist  of  software 
routines  within  the  store-and-forward  IMP's  to 
accumulate  statistics,  periodically  record  snap¬ 
shots,  trace  selected  messages  through  the  net, 
and  to  be  able  to  generate  artificial  traffic. 
These  capabilities  have  been  augmented  by  routines 
at  the  Network  Measurement  Center  to  provide 
selective  control  over  the  data  gathering  func¬ 
tions,  as  well  as,  providing  data  reduction  and 
print-out  routines,  and  more  extensive  artificial 
traffic  generation  facilities.  The  purpose  of 
the  measurement  effort  is  to  provide  insight  Into 
network  behavior,  and  to  support  the  analytic  and 
simulation  modeling  work  being  done  at  UCLA.  In 
this  latter  role,  the  measurements  provide  a 
means  of  verifying,  or  correcting,  models  of  net^- 
work  behavior  by  performing  appropriate  tests  in 
the  actual  network  environment.  The  result  of 
several  such  efforts  will  be  discussed. 

1.  Introduction 

Tne  ARPA  computer  network  interconnects 
approximately  twenty"  different  HOST  computer  sys¬ 
tems  at  sites  across  the  country  for  the  purpose 
of  sharing  resources.^)  Several  of  these  sites 
have  developed  expertise  In  selected  areas  such 
as  graphics,  artificial  intelligence,  and  man- 
machine  communications,  and  serve  as  specializa¬ 
tion  centers  within  the  net  to  provide  particular 
skills  and  hardware /software  resources  related  to 
their  particular  areas  of  specialization.  The 
UCLA  node  in  the  network  serves  ns  one  such  site, 
being  the  Network  Measurement  Center  (NMC) ,  as 
well  as  being  heavily  involved  in  analytic  and 
simulation  modeling  of  network  behavior.  This 
concern  for  analytic  and  empirical  evaluation  of 
network  performance  has  been  an  integral  part  of 
the  network  development,  and  an  early  responsi¬ 
bility  of  the  NMC  was  to  define  the  necessary 
measurement  capabilities  to  be  implemented  in  the 
Interface  Message  Processors  or  IMP’s.**  Subse¬ 
quent  usage  lias  been  made  of  these  measurement 


*  This  research  was  supported  by  the  Advanced 
Research  Projects  Agency  of  the  Department  of 
Defense  under  Contract  Number  DAHC-15-69-C-0285. 

**The  IMP’s  are  small  computers  (modified  Honey¬ 
well  DDP-516s)  which  handle  the  store-end- 
forward  communications  of  the  network,  end  eleo 
have  the  capability  of  collecting  massage 
handling  statistics. 


facilities  to  evaluate  the  network  operational 
status,  to  indicate  areas  where  redesign  may  be 
required,  and  to  provide  Insight  into  the  cause 
and  effect  relationships  of  the  network  behavior. 
This  latter  activity  has  proven  to  be  particularly 
fruitful  In  conjunction  with  the  network  modeling 
efforts  since  the  validity  of  the  models  can 
thereby  be  tested  against  actual  observed  network 
behavior . 

The  IKP-generated  statistics  routines  are  utilized 
In  conjunction  with  programs  at  the  NMC  which 
control  the  data  collection,  process  the  resulting 
measurement  data,  and  generate  desired  levels  of 
artificial  traffic.  These  measurement  tools  will 
be  described  In  the  following  section,  after  which 
some  applications  of  the  measurement  facilities 
will  be  discussed. 

2.  The  Set  of  Measurement  Tools 

The  IMP’s  can  gather  statistics  on  the  net¬ 
work  usage  and  performance  by  means  of  several 
data  gathering  routines  which  can  be  selectively 
enabled  at  appropriate  sites.  These  routines 
Include:3 

•  accumulated  statistics  (counts  and  histo¬ 
grams) 

•  snap-shot  statistics  (queue  lengths  and 
routing  tables) 

•  trace  data  (event  timing  for  message  flow) 

•  status  reports  (activity  and  condition 
Information) 

The  first  three  of  these  routines  have  been 
utilized  extensively  by  the  network  measurement 
center,  while  the  fourth  has  been  utilized 
exclusively  by  the  network  contractor.  Bolt  Beranck 
and  Newman. 

Accumulated  Statistics 

The  accumulated  statistics  routine  has  been 
utilized  more  frequently  than  any  of  the  other 
measurement  tools,  since  it  provides  a  summary 
report  of  the  activity  at  each  node  where  such 
statistics  have  been  enabled.  These  data  reports 
Include  histograms  of  the  lengths  of  HOST- to- IMP 
or  IMP-to-HOST  messages,  and  of  packets  which  were 
transmitted  on  the  modem  output  lines,  as  well  as 
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counts  of  ACK'e*,  RFNM'a,**  Input  errors,  retran- 
nlsslons,  total  wo^ds  sent,  and  other  data  of 
concern.  Each  such  data  message  repreaenta  tha 
activity  over  a  12.8  second  Interval,  this  period 
being  determined  primarily  by  the  concern  for 
counter  overflow.  The  data  Is  sent  to  the  Net¬ 
work  Measurement  Center  as  a  390  byte  message, 
which  1g  then  Inspected  and  either  discarded, 
printed,  or  put  In  a  file  for  further  data  com¬ 
pacting.  When  printed,  the  data  Is  annotated, 
reformatted,  and  In  some  Instances  additional 
values  are  computed,  with  a  resulting  print-out 
as  shown  In  Figure  1.  The  various  fields  of  this 
data  format  are  described  below. 

Part  1  of  the  data  gives  message  alze  statis¬ 
tics  for  both  HOST- to- IMP  and  IMP-to-HOST  data 
transfers.  The  statistics  on  alngle  packet 
messages  are  accumulated  In  a  logarithmic  scale 
histogram,  while  multipacket  traffic  is  recorded 
as  a  uniform  Interval  histogram.  The  average 
number  of  words  In  the  last  packet  of  a  message 
Is  also  given.  In  the  example  shown,  there  were 
a  total  of  200  artlflcally  generated  measagea 
received  from  the  HOSTS,  of  which  208  were  single 
packet  messages  and  twelve  were  two  packets  In 
length. 

The  second  part  of  Figure  1  shows  that  these 
220  messages  were  split  fairly  evenly  between  six 
destinations.  These  sites  were  selected  as 
destinations  for  the  artificial  traffic  to  show 
the  difference  In  delays  when  messages  are  sent 
through  several  store-and-forward  nodes. 

The  round  trip  times  are  measured  from  the 
receipt  of  the  first  packet  of  the  message  at  the 
originating  IMP,  until  the  receipt  of  the  RFNM 
by  this  IMP.  The  data  values  are  accumulated  as 
a  sum  of  round  trip  times  and  a  count  of  the 
number  of  round  trips,  with  the  average  being 
computed  at  the  NMC. 

Part  3  lists  the  activity  for  each  HOST,  both 
real  and  fake,***  the  latter  being  referred  to  as 
GHOST' 8.  The  table  helps  to  determine  which 
HOST's  contributed  to  the  composite  HOST-to-IMP 
behavior  as  recorded  In  Parts  1  and  2. 


*  An  ACK  Is  an  acknowledgement,  i.e.,  a  short 
transmission  between  IMPS  to  Indicate  the  suc¬ 
cessful  reception  of  a  measage  segment 
(referred  to  as  a  packet). 

**  A  RFNM  (Request  For  Next  Message)  la  a  abort 
transmission  which  is  sent  back  to  the  source 
of  a  message  from  the  destination  IMP.  The 
RFNM  is  part  of  a  flow  control  mechaniam  which 
Involves  logical  links  between  source-destina¬ 
tion  pairs,  with  a  given  link  becoming  blocked 
after  use  until  the  receipt  of  a  RFNM. 

***A  Fake  HOST  is  a  background  program  in  the  IMP. 
These  programs  include  the  statistics  genera¬ 
tion  routines,  the  debug  facilities,  and  the 
IMP  console  Teletype  handler. 


Tha  activity  on  tha  Individual  modem  channels 
la  recorded  In  Part  4.  HELLO  and  IKY  (I  Heard 
You)  messages  era  aent  periodically,  and  proper 
counts  Indicate  that  the  llnea  are  active.  (The 
HELLO  aesaage  Is  combined  with  the  routing  table 
update  message.)  The  "IPKTS  RECVD"  total  includes 
these  HELLO  messages,  and  any  ACK'a  and  RFNM's 
that  have  been  received,  as  well  as  the  number  of 
actual  message  packets  that  have  been  received. 
Other  data  Include  the  number  of  RFNM's  sent,  the 
total  number  of  data  words  sent,  the  number  of 
times  an  arrival  found  the -free  storage  list 
empty,  and  In  such  cases,  the  number  of  times 
that  an  unacknowledged  store-and-forward  packet 
was  discarded  to  free  an  input  buffer. 

The  final  section  of  Figure  1  shows 
logarithmic  scale  histograms  of  packet  lengths 
on  each  modem  channel.  Only  the  message  content 
of  the  transmlsaiona  are  recorded  In  these  histo¬ 
grams,  i.e.,  they  do  not  include  the  HELLO,  IHY, 
ACK,  or  RFNM  traffic.  The  choice  of  the  loga¬ 
rithmic  acale  was  due  to  a  priori  expectations 
on  the  length  distribution. 

The  various  portions  of  the  accumulated  data 
can  be  utilized  together  to  develop  a  reasonably 
good  picture  of  the  activity  over  the  12.8  second 
Interval.  In  the  example  of  Figure  1  we  know  the 
total  number  of  messages  from  the  HOST,  their 
final  destinations,  and  their  average  round  trip 
times.  The  HOST-IMP  histograms  give  us  informa¬ 
tion  of  the  message  size  distribution,  and  in  the 
modem  channel  histograms  give  similar  data  for 
the  packet  size  distribution  as  well  as  channel 
utilization  information.  In  this  example,  the 
channel  statistics  show  that  more  packets  were 
sent  via  SRI  than  was  expected  from  the  routing 
table  at  the  UCLA  IMP,  which  would  normally  send 
about  half  of  these  packets  via  RAND.  Such  data 
can  lead  the  experimenter  to  probe  further  into 
the  cause  of  such  behavior,  but  to  do  so,  he  needs 
the  routing  table  Information  which  is  contained 
in  another  set  of  measurements  called  snap-shots. 

Snap-Shot  Statistics 

Fach  snap-shot  statistics  message  contains 
the  queue  lengths  and  the  routing  table  informa¬ 
tion  for  that  particular  IMP  and  that  particular 
instant  of  time.  These  values  are  recorded  and 
sent  at  0.82  second  intervals,  this  time  period 
being  a  reasonable  compromise  between  the  desire 
to  see  a  time-sequence  of  state  changes,  and  the 
conflicting  desire  to  reduce  the  artifact*  caused 
by  sending  the  statistics  too  frequently.  Like 
all  of  the  other  measurement  tools,  the  snap-shota 
can  be  enabled  or  disabled  at  each  Individual 
IMP. 


'  ■ rtifact  in  measurement  data  refers  to  any 
effect  that  resulta  In  a  difference  between  the 
actual  system  variables  (as  they  would  occur 
without  any  measurements)  and  the  observed  data. 
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Figure  !•  A  Typical  Accumulated  Statistic  Print-Out. 
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race  Statistic; 

The  trace  capability  allova  one  to  literally 
>llow  a  message  through  the  network,  and  to  learn 

■  the  route  which  It  takes  and  the  delays  which 
encounters.  However,  It  is  one  of  the  more 

•mpllcated  of  the  measurement  data  formats  to 
-•scribe  because  the  interpretation  of  the  data 
spends  on  the  function  of  the  IMP  handling  the 
ssage,  l.e.,  either  source,  store-and-forward, 

'  destination,  as  well  as  depending  on  the  possl- 
llty  of  retransmissions. 

A  typical  store  and  forward  situation  will, 

■en  traced,  result  in  a  record  of  (1)  the  time 
arrival,  (2)  the  time  at  which  the  message  is 
ocessed,  l.e.,  put  on  an  output  queue,  (3)  the 
me  at  which  transmission  is  Initiated,  and  (4) 
e  time  when  the  ACK  is  received.  In  the  case  of 
retransmission,  this  latter  time  is  the  time  of 
transmission,  and  is  so  indicated  by  a  tag  bit 
the  data  block.  At  the  destination  IMP,  this 
urth  word  is  the  time  at  which  the  IMP-to-HOST 
ansmlsslon  waa  completed.  All  times  are 
corded  in  terms  of  a  clock  with  a  100  psec. 
solution. 

The  other  information  in  the  trace  data  mes- 
ge  includes  the  output  channel  (if  on  a  modem 
annel),  the  HOST  number,  or  the  "fake  HOST" 
ciber,  as  well  as  the  entire  header  of  the  packet, 
e  header  contains  the  source,  the  destination, 
e  link  number,  the  message  number,  the  packet 
nber,  and  the  priority /non-priority  status. 

tlflclal  Traffic  Generation 

The  measurement  facilities  of  the  network  are 
tmented  by  artificial  traffic  generation  rou- 
ie8  both  in  the  IMPs  and  at  the  NMC.  Such 
•.ificlal  traffic  was  helpful  in  earlier  "shake- 
vn"  tests  of  the  network  operation,  and  more 
■ently  has  been  a  necessary  tool  in  our  work 
bating  measurements  and  modeling. 

Each  IMP  can  generate  artificial  traffic 
islsting  of  fixed  length  messages  that  are 
-lodlcally  sent  on  a  single  link  at  a  selected 
ie  Interval,  but  such  traffic  is  not  sufficient 
simulate  congestion  at  the  IMP’s  due  to  the 
>w  control  mechanism  which  is  built  into  the 
lk/ltl'NM  operation.  A  more  powerful  artificial 
ifflc  generator  was  therefore  developed  oh  purt 
the  NMC  capabilities,  and  lncladcH  the  ability 
send  messagea  on  op  to  61  links,  to  multiple 
uInntlonH,  with  either  fixed  or  random  message 
■'gibs  and  Inter-t  rans'ii  I  salon  times.  The  traffic 
I  ell  waa  lit  Mixed  In  the  example  of  Figure  I  was 
"listed  by  llila  latter  facility  utilizing  the 
idom  variable  features  as  indicated  by  thu 
■iBSgo  length  histograms. 

■  Measurements  Related  to  Network  Applications 

Applications  of  the  network  were  initially 
n'ted  by  the  lack  of  an  established  protocol 
t  dOST-to-HOST  transmission,  although  several 
tea  made  use  of  the  network  by  establishing 


special  purpose  protocol  subsets  for  handling 
their  particular  functions  of  interest  such  as 
interactive  work,  file  transmissions,  or  graphics 
display  functions.  Rather  than  summarizing  the 
measurement  facilities  relative  to  a  number  of 
these  application  areas,  we  will  present  a  more 
detailed  accounting  of  the  measurements  that  were 
taken  on  the  activity  of  one  set  of  users. 

The  activity  of  interest  involved  the  use 
of  the  DEC  PDP-10  at  the  University  of  Utah  and 
the  XDS  940  at  SRI,  with  the  general  nature  of 
the  interaction  being  as  follows.  A  file  would 
be  sent  from  SRI  to  Utah,  and  then  a  number  of 
interactive  debug  operations  would  be  performed, 
with  transmission  from  the  XDS  940  to  the  PDP-10 
being  on  a  character-by-character  basis,  and 
responses  being  handled  on  a  line  at  a  time  basis. 
The  files  were  transmitted  in  blocks  of  4608  bits, 
which  was  a  convenient  multiple  of  the  24-bit 
word  length  of  the  XDS  940  and  the  36-blt  word 
length  of  the  PDP-10,  and  also  provided  the 
desired  block  size  of  128  words  for  the  PDP-10. 

The  established  protocol  involved  several 
special  characters  to  indicate  the  message  type 
and  length,  such  that  a  single  character  message 
required  five  IMP-words  of  transmitted  data. 

Due  to  these  character-by-character  and  file 
transmission  protocol  conventions,  almost  all  of 
the  SRI-to-Utah  transmissions  consisted  of  either 
5-word  messages  or  5-packet  messages  respectively, 
as  shown  in  the  sample  print-out  of  Table  1. 

Each  line  corresponds  to  one  12.8  second  statis¬ 
tics  message,  with  the  time-of-day  being  deter¬ 
mined  by  adding  12.8  second  increments  to  the 
starting  time  since  the  data  values  were  taken 
consecutively.  The  sequence  of  events  which  is 
shown  includes  a  10-word  "file  name"  message, 
the  file  transmission  of  67  5-packet  messages 
plus  a  4-packet  file  fragment,  an  "end  of  file" 
message,  and  a  sequence  of  interactive  traffic. 
This  particular  file  transmission  consisted  of 
approximately  39,300  bytes  and  occurred  over  a 
25.6  second  period  for  an  average  bandwidth  usage 
of  about  12.5  Kbits/second.  The  maximum  file 
transmission  bandwidth  observed  during  uny  12.8 
second  interval  was  one  in  which  73  5-pncket 
messages  were  transmitted.  Each  message  was 
the  standard  576  byte  record,  for  a  total  band¬ 
width  usage  of : 

26.3  Kh/sec  for  data  ("effective"  data  rate") 

0.3  Kh/sec  for  marking,  pudding,  and  protocol 
format 

1,_9  Kh/sec  for  network  header,  checksum,  etc, 

30.5  Kh/sec  total  hinnlwldt li  used. 

Trsoiniilniilon  occi  rreil  during  the  entire 
interval  since  thu  file  t  riiniim  I  its  I  on  wan  measured 
over  three  12.8  sec.  Intervals,  mid  this  was  the 
center  Interval.  The  average  round  trip  time 
for  a  message  was  159  msec,  (measured  from  the 
time  the  source  IMP  receives  the  first  packet 
until  it  received  the  HPNM).  The  minimum  round 
trip  time  would  be  about  138  msec,  based  on  the 
time  for  serial  transmissions  and  propagation 
delays.  The  extra  delay  of  159  minus  138,  or 
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TABLE  1 


A  SUBSET  CF  YHE  ACCUMULATED  STATISTICS  MEASUREMENT 
OF  THE  SRI-WAH  TRAFFIC 
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*  MEASURED  IN  TEEMS  CF  IMP  WORDS 

**  MEASURED  IN  TEAMS  OF  THE  NUMBER  OF  PACKETS  PER  MESSAGE 


21  msec.,  was  due  to  the  delays  in  the  PDP-10 
acceptance  of  messages.  The  effective  round 
trip  cycle  must  include  an  additional  12  msec, 
for  HOST  processing  of  the  RFNM  and  for  the 
leader  and  first  packet  transmission  from  the  HOST 
to  the  IMP.  Therefore,  the  maximum  single  link 
data  rate  Is  approximately  one  576  byte  record 
every  150  msec,  or  30.7  Kb/sec.  of  actual  data, 
compared  with  the  observed  maximum  effective  data 
rate  of  26.3  Kb/sec.  A  similar  calculation  using 
the  measured  average  round  trip  time  and  the 
resulting  effective  data  rate  indicates  that  about 
4  to  5  msec,  are  required  for  HOST  processing  of 
the  RFNM  and  setting  up  the  next  transmission. 

The  round  trip  time  required  for  messages 
between  the  two  HoST's  was  plotted  in  Figure  2, 
as  a  function  of  message  length  L.  Each  point  of 
this  scatter  plot  represents  one  interactive 
message,  l.e.,  a  one-packet  message.  The  scatter¬ 
ing  is  an  indication  of  the  frequency  of  occur¬ 
rence  of  both  message  length  variations  and  round 
trip  delay  variations.  A  definite  lower  limit  of 
delay  was  observed  as  indicated  by  the  dashed 
line,  and  this  limiting  delay  function  Is  Just 
what  would  be  expected  based  on  estimated  values 
of  the  various  components  of  delay.  In  this 
minimum  delay  case,  there  is  no  queueing  time 

/ 


such  that  the  expected  delay  consists  of: 


.  Serial  trans.  at  Bource  IMP  "  2.0  +  0.32L  msec. 

•  Propagation  &  modem  delay  ■  7.5  msec. 

•  Processing  time  at  dest.  »  O.i  msec. 

.  Serial  IMP-HOST  trans.  ■  0.3  +  0.16L  msec. 

»  Serial  transmission  of  RFNM  -  2.0  msec. 

•  Propogation  and  modem  delay  »  7.5  msec. 

•  Processing  of  RFNM  ■  0.2 _ msec ■ 

Total  20.0  +  0.48L  msec. 


This  estimated  total  delay  agrees  very  well  with 
the  experimental  minimum  delay  values  as  shown 
in  Figure  2.  The  other  points  in  the  figure 
represent  messages  whose  delay  values  exceeded  the 
minimum  due  to  queueing  delays  and/or  HOST  delays 
in  taking  the  message  from  the  IMP. 

4.  Measurements  to  Determine  Network  Performance 
and  Limiting  Values 

Several  experiments  were  conducted  to  evaluate 
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Round-trip  delay  (SRI-Utah) 
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Figure  2.  Round-Trip  Delay  Measurements 
of  the  SRI-Utah  Traffic. 

various  aspects  of  the  network  performance  at 
different  traffic  levels,  and  to  determine  para¬ 
meter  sensitivities  and  limiting  values.  The 
following  example  is  typical  of  these  experiments, 
and  involves  the  measurement  of  the  through-put 
between  two  nodes  with  the  effect  of  alternate 
routing  being  considered  by  both  analytic  and 
measurement  techniques.  The  purpose  of  the 
example  is  to  show  how  the  two  techniques  are 
complementary,  with  the  measurements  providing 
the  insight  needed  to  improve  and  correct  the 
analytic  model  of  the  network  behavior. 

The  initial  through-put  tests  were  conducted 
by  sending  artificial  traffic  from  the  NMC  at 
UCLA  to  the  discard  "fake  HOST"  in  the  UCSB  IMP, 
and  measuring  the  resulting  through-put  as  a 
function  of  the  number  of  links  utilized  in  the 
transmission. 

The  communication  lines  between  the  IMP's 
have  a  bandwidth  of  50  Kbits/second,  which 
establishes  an  upper  bound  on  the  single  path 
through-put ,  However,  the  ovarhoed  associated 
with  each  message  limits  the  total  through-put 
to  some  lesser  value  as  shown  in  Figure  3.  The 
same  figure  also  indicates  the  measured  through¬ 
put  for  the  case  of  alternate  routing  which 
exceeds  the  50  Kbit/second  value  due  to  the 
bifurcation  of  the  traffic  flow.  In  all  cases 
the  messages  were  sent  at  the  RFNM  driven  rate, 
i.e.,  as  soon  as  a  link  would  become  unblocked. 


Figure  3.  Through-put  Measurements  Between  UCLA 
and  the  Neighboring  UCSB  IMP 

effective  packet  overhead  was  minimized  (relative 
to  the  packet  size) .  The  maximum  through-put  for 
such  messages  was  measured  to  be  38.6  Kbits/ 
second,  which  agrees  quite  well  with  the 
theoretical  value  of  39.0  Kbits/second  (based  on 
20.0  msec,  of  useful  data  transfer  and  5.7  msec, 
for  the  packet  overhead  and  the  acknowledgement 
for  the  RFNM) .  This  maximum  through-put  is  not 
reached  for  the  case  of  a  single  link  due  to  the 
quiescent  period  while  waiting  for  the  RFNM  to 
return.  This  delay  is  about  13.5  msec.,*  which 
adds  to  the  "non-productive"  transmission  period, 
and  results  in  an  expected  single  link  through¬ 
put  of  25.5  Kbits/second.  (The  measured  value 
was  22.2  Kbits/second.)  The  through-put  for  the 
one-packet  message  case  can  be  seen  to  Increase 
abruptly  when  the  transition  is  made  from  six  to 
seven  traffic  generators,  due  to  alternate  rout¬ 
ing  via  the  SRI  path. 

The  single  link  through-put  for  the  full  8- 
packet  message  case  was  seen  to  be  approximately 
equal  to  the  aaturation  value  of  the  single  packet 
case,  l.e.,  38.5  Kbits/second.  This  equality  is 
due  to  the  canceling  effects  of  fewer  RFNM's  to 
acknowledge,  and  the  short  quiescent  delay  for 
the  single  link  RFNM.  No  alternate  routing  was 
observed  until  the  two  generator  test,  above  which 
the  traffic  split  evenly  between  the  direct  UCLA- 
UCSB  and  the  UCLA-SRI-UCSB  routes. 


Two  different  message  sizes  were  Included  in 
the  above  test.  In  the  first  case,  the  messages 
were  one  full  packet  in  length  so  that  the 


*  Based  on  a  measured  round  trip  time  from  UCLA 
to  l£SB  of  22.8  msec,  for  a  320  bit  message  and 
the  known  service  time  of  9.3  msec,  for  a  message 
of  that  length. 
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The  saturation  effect  for  the  8-packet 
message  case  occurred  at  a  lower  bandwidth  than 
was  expected,  since  the  direct  and  alternate 
route  paths  should  each  be  able  to  deliver 
messages  with  an  effective  rate  of  about  43.5 
Kbits/second  (which  is  higher  than  the  single- 
packet  through-put  since  there  are  fewer  RFMM's  to 
acknowledge).  The  measured  limiting  value  of 
about  70  Kbits/second  was  therefore  Investigated 
further  to  try  to  determine  the  cause  of  this 
behavior.  Several  possible  reasons  for  this 
discrepancy  were  considered  and  described  below, 
since  they  represent  the  possible  limiting  factors 
in  such  transmissions  and  indicate  the  manner 
in  which  the  statistics  gathering  tools  can  be 
utilized  to  determine  the  cause  of  anomalies. 

a‘  of  store-and-forward  buffers;  In 

order  to  obtain  the  maximum  through-put,  there 
must  he  a  sufficient  quantity  of  store-and-forward 
buffers  to  be  able  Co  create  a  queue  that  is  long 
enough  to  keep  the  channel  busy  during  the  entire 
routing  update  interval  (which  was  thought  to  be 
about  520  msec.).  Therefore,  with  a  packet 
service  time  of  23.0  msec.,  the  queue  would  have 
to  build  up  to  a  total  of  about  23  buffers.  Since 
we  understood  that  there  was  a  total  of  25  store- 
and-forward  buffers,  this  did  not  appear  to  be 
the  limiting  factor.  Snap-shot  measurements  also 
showed  that  typically  18  to  20  store-and-forward 
buffers  were  in  use,  which  seemed  to  substantiate 
this  conclusion. 


of  25.)*  If  we  reconsider  the  effect  of  the 
finite  buffering  with  these  new  parameter  values, 
we  find  that  the  first  channel  will  have  approxi¬ 
mately  18  packets  to  serve  when  the  routing 
change  occurs,  and  at  23.0  msec,  of  service  time 
per  packet,  that  channel  will  remain  husy  for 
about  420  msec.**  Since  the  next  routing  up, late 
will  occur  after  640  msec.,  the  first  channel 
will  be  inactive  for  220  of  the  640  msec.,  for 
an  effective  through-put  contribution  of: 

through-put  -  ~  .  (43.5  Kblts/sec.) 

“  28.5  Kbits/sec. 

for  a  total  estimated  through-put  of  72  Kbits/ 
second,  compared  to  the  measured  70  Kbits/second, 
which  is  a  reasonable  match  between  the  theory 
and  measurement.  ’ 

This  test  activity  was  described  in  some 
detail  to  indicate  how  the  various  measurement 
tools  can  be  utilized  to  reconcile  differences 
between  the  expected  and  experimental  data  values. 
However,  it  also  points  out  an  equally  important 
aspect  of  the  experimentation;  namely  that  tinse 
people  involved  in  the  network  measurement  a-.d 
modeling  efforts  must  be  aware  of  any  changes  in 
the  network  parameters  if  they  are  to  be  able  to 
properly  Interpret  the  test  results. 

6.  Summary 


b’  Possible  bandwidth  reduction  effects: 

The  effective  transmission  bandwidth  can  be 
reduced  by  a  number  of  effects  including  line 
errors,  lack  of  reassembly  buffers,  and  routing 
loops.  However,  measurements  indicated  that  none 
of  these  effects  were  causing  a  significant 
through-put  reduction. 


c ■  Interface  and  HOST  computer  system 
-Limitations:  The  transmission  capabilities  of 
HOST  and  HOST-IMP  interface  were  verified  by 
directing  the  artificial  traffic  to  the  locdl 
(UCLA)  discard  "fake  HOST"  with  an  observed 
through-put  cf  91.5  Kbits/second,  which  far 
exceeded  the  UCLA-to-UCSB  saturation  value. 


<*•  Trace  measurements  to  determine  the 
tietailed  behavior:  Since  the  accumulated  statis¬ 
tics  aid  the  snap-shots  both  indicated  that  the 
limitation  had  to  be  in  the  UCLA  IMP,  a  detailed 
investigation  of  the  packet  handling  behavior  was 
made  by  tracing  all  of  the  packets  at  the  UCLA 
IMP  for  a  brief  instant  of  time.  (At  the  traffic 
rates  involved,  about  100  packets/second  were 
generated  and  each  packet  resulted  in  a  separate 
trace  message.)  This  trace  data  indicated  two 
errors  in  the  values  of  the’  IMP  parameters  that 
had  been  used  in  the  calculations  of  part  (a) 
above.  First  the  routing  updates  were  found  to 
occur  at  approximately  640  msec,  intervals  Instead 
of  the  520  msec,  that  .his  parameter  was 
previously  understood  to  be,  and  secondly,  the 
upper  limit  on  the  store-and-forward  buffers  on 
an  output  queue  must  be  spproxiaately  20  (instead 


The  measurement  facilities  which  have  been 
included  in  the  design  of  the  network  IMPs  have 
been  very  useful  for  network  performance  evalua¬ 
tion,  and  for  gaining  insight  to  determine  the 
sensitivities  and  limiting  effects  of  the  network 
behavior.  This  insight  has  also  been  valuable 
for  improvement  of  analytic  models,  by  comparing 
the  model  behavior  to  the  actual  measured  network 
behavior. 
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*  A  subsequent  check  with  BBN  showed  that  the 
actual  routing  update  interval  is  655  msec. 

The  upper  limit  on  store-and-forward  buffers  is 
25  (octal)  which  is  21  (decimal). 

**  The  other  two  store-and-forward  packets  are 
assumed  to  be  required  for  the  SENT  queue. 
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